Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(java): typed-decisions end to end, and the pairing that answers anyway

Adeshina Lasisiba028702ebac
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index b8a8ac0c..7b5a117d 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -333,15 +333,23 @@ jobs:
# and is the whole runtime of this lane. Running them in one job made a 13-second check wait on
# a 40-minute export.
#
- # NOT split per checkpoint, which is where the .NET lane splits. Two reasons, both measured.
- # The fixtures are not per-checkpoint documents -- `tokenizer_ids.json`, `sequences.json` and
- # `decode.json` each carry BOTH checkpoints' sections in one file, so a job holding one of them
- # cannot regenerate any of the three and `--strict` fails by construction; that is what
- # happened on the first run of this lane. And the Java tests assert that both checkpoints' cases
- # are present, a floor that exists because a @TestFactory whose fixture key emptied used to
- # generate zero assertions and still exit 0. Splitting per checkpoint would mean deleting that
- # floor to buy nothing: the two tokenizers together are a few megabytes, so there is no export
- # to parallelise away.
+ # The GRAPHLESS cell is not split per checkpoint, which is where the .NET lane splits. Two
+ # reasons, both measured. The fixtures it owns are not per-checkpoint documents --
+ # `tokenizer_ids.json`, `sequences.json` and `decode.json` each carry BOTH checkpoints'
+ # sections in one file, so a job holding one of them cannot regenerate any of the three and
+ # `--strict` fails by construction; that is what happened on the first run of this lane. And
+ # the Java tests assert that both checkpoints' cases are present, a floor that exists because a
+ # @TestFactory whose fixture key emptied used to generate zero assertions and still exit 0.
+ # Splitting it per checkpoint would mean deleting that floor to buy nothing: the two tokenizers
+ # together are a few megabytes, so there is no export to parallelise away.
+ #
+ # The WEIGHT-BEARING cells are one per checkpoint, and have to be. A graph is traced from
+ # exactly one checkpoint and carries its weights, so `predict.json` (multilingual) and
+ # `typed_decisions.json` (typed-decisions) need two graphs; a cell holding both would pay both
+ # exports serially, which is the cost this whole split exists to avoid. Each such cell declares
+ # the checkpoint it traces in `graph-model`, which is also what keys its export cache and which
+ # of the two graph variables gets a path -- so adding a third never silently reuses the second's
+ # graph.
parity:
name: jvm parity (${{ matrix.name }})
runs-on: ubuntu-latest
@@ -359,7 +367,7 @@ jobs:
# 40-minute export for nothing.
- name: tokenizer
checkpoints: english multilingual
- graph: false
+ graph-model: "" # nothing traced here
families: >-
--only tokenizer_ids.json --only sequences.json --only decode.json
--only decode_text.json --only window_plan.json --only predict_long.json
@@ -385,15 +393,44 @@ jobs:
# assertion in TokenDecoderTest and the `cases.size() >= 23` corpus floor beside it.
min-tests: 489
timeout: 25
- # The one cell that needs the traced graph, and the only checkpoint the end-to-end
- # golden was recorded from.
+ # Weight-bearing: the multilingual end-to-end golden, traced from that checkpoint.
- name: predict
checkpoints: multilingual
- graph: true
+ graph-model: multilingual
families: --only predict.json
tests: --tests com.convaiinnovations.laya.PredictParityTest
min-tests: 9
timeout: 75
+ # Weight-bearing, and a SEPARATE cell rather than a second family in the one above.
+ # Three reasons, in order of how much they cost:
+ #
+ # * A graph is traced from one checkpoint. `predict.json` is recorded against
+ # multilingual and `typed_decisions.json` against typed-decisions, so one cell
+ # holding both would export two graphs and run two exports serially -- the 40
+ # minutes this matrix was split to avoid, paid twice in one cell instead of once
+ # in each of two that run in parallel.
+ # * The export cache is keyed on `graph-model`, so each checkpoint's graph is
+ # invalidated by its own revision bump. A merged cell would key two graphs under
+ # one entry and re-export both whenever either moved.
+ # * These two goldens fail for unrelated reasons. A merged cell reports one status
+ # for both, and the first thing a reader of a red parity lane needs to know is
+ # which checkpoint moved.
+ #
+ # Cost, measured locally with the Hub blobs already cached: 846 MB of checkpoint, a
+ # 39 s ONNX export producing 3.6 MB of graph beside 1.7 GB of external weights, 103 s
+ # to record the family and 109 s for the 13 tests. On a cold runner the download is
+ # the open variable and is why this carries the same 75-minute ceiling as `predict`.
+ - name: typed-decisions
+ checkpoints: typed-decisions
+ graph-model: typed-decisions
+ families: --only typed_decisions.json
+ tests: --tests com.convaiinnovations.laya.TypedDecisionsParityTest
+ # Measured locally: 13 -- 8 single-state cases, 3 batch configs, the config identity
+ # check and the workflow-signature check. Exact rather than padded down, because the
+ # two non-factory tests are what catch a wrong checkpoint/graph pairing and a margin
+ # of 2 would let both be dropped with this floor still green.
+ min-tests: 13
+ timeout: 75
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
# Validate the wrapper jar BEFORE anything can execute it. See this file's
@@ -435,16 +472,20 @@ jobs:
# the revision stamp now makes safe) and accepting a multi-gigabyte entry for the
# weight-bearing cell; neither is this change's call to make.
- name: Cache the exported graph
- if: ${{ matrix.graph }}
+ if: ${{ matrix.graph-model != '' }}
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: laya-java/.work/onnx
- key: laya-java-onnx-${{ runner.os }}-${{ matrix.name }}-${{ hashFiles('laya-java/scripts/prepare_checkpoint.py', 'scripts/export_onnx.py', 'laya/common.py', 'laya-dotnet/tools/requirements-regen.txt') }}
+ key: laya-java-onnx-${{ runner.os }}-${{ matrix.graph-model }}-${{ hashFiles('laya-java/scripts/prepare_checkpoint.py', 'scripts/export_onnx.py', 'laya/common.py', 'laya-dotnet/tools/requirements-regen.txt') }}
+ # A graph is exported for the ONE checkpoint this cell's golden was recorded from, and
+ # every other checkpoint it needs is fetched `--no-graph`. Keyed on the name rather than on
+ # a boolean, because `graph: true` could not say WHICH checkpoint to trace, and a cell that
+ # names two would otherwise have traced both.
- name: Prepare the checkpoints this cell owns
run: |
for c in ${{ matrix.checkpoints }}; do
- if [ "${{ matrix.graph }}" = "true" ]; then
+ if [ "$c" = "${{ matrix.graph-model }}" ]; then
python laya-java/scripts/prepare_checkpoint.py --checkpoint "$c"
else
python laya-java/scripts/prepare_checkpoint.py --checkpoint "$c" --no-graph
@@ -458,13 +499,19 @@ jobs:
- name: Regenerate this cell's fixtures from Python
env:
LAYA_FIXTURE_CHECKPOINTS: ${{ github.workspace }}/laya-java/.work/checkpoints
- LAYA_ONNX_GRAPH: ${{ matrix.graph && format('{0}/laya-java/.work/onnx/multilingual/laya.onnx', github.workspace) || '' }}
+ # One variable per golden, and only this cell's is populated. They cannot share one:
+ # a graph carries the weights and nothing in it says which checkpoint traced it, so
+ # `predict.json` pointed at the typed-decisions graph would re-record itself from the
+ # wrong weights and compare them against themselves. Measured: the english checkpoint
+ # against the typed graph raises nothing and answers to the same last digit.
+ LAYA_ONNX_GRAPH: ${{ matrix.graph-model == 'multilingual' && format('{0}/laya-java/.work/onnx/multilingual/laya.onnx', github.workspace) || '' }}
LAYA_PREDICT_MODEL: multilingual
+ LAYA_TYPED_ONNX_GRAPH: ${{ matrix.graph-model == 'typed-decisions' && format('{0}/laya-java/.work/onnx/typed-decisions/laya.onnx', github.workspace) || '' }}
run: python laya-java/scripts/gen_fixtures.py --strict ${{ matrix.families }}
# The pre-tokenizer's letter and number classes are probed out of the reference tokenizer,
# so this is the one generator gate that needs a checkpoint. It belongs to the cheap cell.
- name: Pre-tokenizer class tables match the reference
- if: ${{ !matrix.graph }}
+ if: ${{ matrix.graph-model == '' }}
run: >
python laya-java/scripts/gen_pretokenizer_tables.py --check
--tokenizer laya-java/.work/checkpoints/english/tokenizer/tokenizer.json
@@ -479,7 +526,8 @@ jobs:
working-directory: laya-java
env:
LAYA_CHECKPOINTS: ${{ github.workspace }}/laya-java/.work/checkpoints
- LAYA_ONNX_GRAPH: ${{ matrix.graph && format('{0}/laya-java/.work/onnx/multilingual/laya.onnx', github.workspace) || '' }}
+ LAYA_ONNX_GRAPH: ${{ matrix.graph-model == 'multilingual' && format('{0}/laya-java/.work/onnx/multilingual/laya.onnx', github.workspace) || '' }}
+ LAYA_TYPED_ONNX_GRAPH: ${{ matrix.graph-model == 'typed-decisions' && format('{0}/laya-java/.work/onnx/typed-decisions/laya.onnx', github.workspace) || '' }}
run: ./gradlew --no-daemon test ${{ matrix.tests }}
# A checkpoint-backed factory that aborts is not a pass. With this cell's checkpoints present
# nothing it runs may abort, and the floor catches a suite that silently stopped registering
diff --git a/laya-java/CHANGELOG.md b/laya-java/CHANGELOG.md
index ef9dba65..13de5510 100644
--- a/laya-java/CHANGELOG.md
+++ b/laya-java/CHANGELOG.md
@@ -67,14 +67,26 @@ see [Installing](README.md#installing).
### Testing
- 769 tests, 626 of which need no model at all.
+- `typed-decisions` is covered end to end by `TypedDecisionsParityTest` against
+ `fixtures/typed_decisions.json`: **13 tests** with the checkpoint present — eight single-state
+ cases spanning all four fine-tuned workflow signatures, three batch configurations, and two
+ checks on the weights themselves. Four of the 13 register without a checkpoint and one of those
+ passes there, since the workflow-signature check reads only the fixture.
+- That second pair of checks is not ceremony. `typed-decisions` is fine-tuned from `english` and
+ ships its `tokenizer.json` byte for byte and the same encoder, so opening the english
+ checkpoint against the typed graph raises nothing and **seven of the eight** single-state cases
+ still pass. The eighth is `over-budget`, which only truncates differently because `max_len` is
+ 512 there and 1024 here; the config check is what names the problem, and it runs before any
+ probability is compared.
- Every expectation is **generated from the Python package**, never written by hand:
`laya-java/fixtures/*.json`, regenerated by `scripts/gen_fixtures.py` and gated by `--check` in
CI so a change to `laya/` that moves a recorded contract shows up here rather than as drift.
- The compiled-in Unicode tables are checked by a **sha256 over all 1,114,112 code points** per
property, so they are proven exhaustively rather than on the cases someone thought to write.
-- Six CI lanes: build plus model-free tests, the same tests on JDK 17, 21 and 24 — three
- different Unicode versions, 13.0, 15.0 and 16.0 — and two parity cells split by whether they
- need a traced graph.
+- Seven CI lanes: build plus model-free tests, the same tests on JDK 17, 21 and 24 — three
+ different Unicode versions, 13.0, 15.0 and 16.0 — and three parity cells split by whether they
+ need a traced graph, and then by which checkpoint that graph was traced from. A graph is
+ traced from exactly one checkpoint and carries its weights, so the cells cannot share one.
- `-PtestJavaVersion` runs the tests on a different JDK from the one the classes are compiled
for, because the toolchain pins the compiler to 17 and a lane that merely installs another JDK
tests the same thing twice.
@@ -89,9 +101,10 @@ see [Installing](README.md#installing).
- Not implemented: hooks, `predictLong`, structured `decide`, the `laya-java-client` HTTP module,
Android.
-- `typed-decisions` is routable — `Router` knows it and resolves its aliases — but has no recorded
- end-to-end fixtures. The routing decision is tested; a forward pass against that checkpoint is
- not.
+- Only `invoice_processing`'s trained question schema is in this repository, so the other three
+ typed-decisions workflows are recorded by their question-id signature with instruction text
+ written for the fixture. The ids are what routing reads and the text is recorded verbatim, so
+ the forward pass is real; the trained wording is not claimed to be.
- The softmax is computed in `double` where Python computes it in float32, as `laya-ts` and
`laya-dotnet` also do. Over 200,000 random logit rows the four-decimal probabilities differ on
88 of them. Rounding, by contrast, is exact: `BigDecimal` at scale 4 with `HALF_EVEN` is
diff --git a/laya-java/MODELS.md b/laya-java/MODELS.md
index 06ec071a..6905e492 100644
--- a/laya-java/MODELS.md
+++ b/laya-java/MODELS.md
@@ -54,11 +54,44 @@ checkpoint the Python package knows about is one this can fetch.
`typed-decisions` is never selected automatically. `Router` knows it, and the aliases `typed`,
`typed_decisions`, `decisions` and `laya-typed-decisions` all resolve to it, but reaching it takes
an explicit `task` or `autoTaskDetection` — it is fine-tuned on four specific workflows and is the
-wrong silent default. **It has no recorded end-to-end fixtures in this port yet**: the routing
-decision is tested, the forward pass against that checkpoint is not.
+wrong silent default.
-A prepared checkpoint is about **680–810 MB** on disk, and the fused multilingual graph is
-**1.29 GB** (`laya.onnx` is a 2.8 MB graph beside a `laya.onnx.data` of external weights). Budget
+Its forward pass **is** recorded, in `fixtures/typed_decisions.json`, and asserted by
+`TypedDecisionsParityTest` in the `jvm parity (typed-decisions)` cell: eight single-state cases
+covering all four workflow signatures, three batch configurations, and the two checks that the
+weights under test are the right ones. Regenerate it with:
+
+```bash
+python laya-java/scripts/prepare_checkpoint.py --checkpoint typed-decisions
+LAYA_TYPED_ONNX_GRAPH=laya-java/.work/onnx/typed-decisions/laya.onnx \
+ python laya-java/scripts/gen_fixtures.py --only typed_decisions.json
+```
+
+`LAYA_TYPED_ONNX_GRAPH`, not `LAYA_ONNX_GRAPH`, and the two must never be crossed. A graph
+carries the weights and nothing in it names the checkpoint it was traced from, so a wrong pairing
+answers instead of failing — `typed-decisions` is fine-tuned **from** `english` and ships its
+`tokenizer.json` byte for byte and its ModernBERT-large encoder. Measured: the english checkpoint
+opened against the typed-decisions graph raises nothing and returns
+`{refund: 0.9077, replace: 0.0923}`, the same digits as the correct pairing, and `model` reads
+`laya-rl-agent-onnx` either way. Measured with the english checkpoint symlinked in as
+`typed-decisions/`: **seven of the eight** single-state cases still pass, and the eighth,
+`over-budget`, fails only because `max_len` is 512 there and 1024 here so the state truncates at
+a different point. What names the problem is the `config` section of the golden — `max_len` 1024
+against 512, `head_max_len` 256 against 192, base temperatures `[1.0148, 1.0374, 1.0575]` against
+`[1.6369, 1.2514, 1.9834]` — which the test asserts before it compares a probability. Only the
+multilingual checkpoint fails loudly against that graph, and for an unrelated reason: a 235292
+token id against a 50368-row embedding, which ONNX Runtime rejects at the `node_embedding` Gather.
+
+Of the four workflows, only `invoice_processing`'s trained question schema is in this repository
+(`examples/32_typed_decisions_workflow.py`). The other three are known by their question-id
+signatures alone, from `laya.router._TYPED_DECISION_WORKFLOWS`, so the golden carries those ids
+with instruction text written for the fixture. The ids are what the routing decision reads and the
+text is recorded verbatim, so both halves are real; the trained wording is not claimed.
+
+A prepared checkpoint is about **680–850 MB** on disk, and the fused multilingual graph is
+**1.29 GB** (`laya.onnx` is a 2.8 MB graph beside a `laya.onnx.data` of external weights). The
+typed-decisions checkpoint is **846 MB** and its graph is **1.69 GB** (3.6 MB beside the external
+weights); the export took **39 s** locally, and recording the family another **103 s**. Budget
accordingly; the export is the slow part.
---
diff --git a/laya-java/fixtures/typed_decisions.json b/laya-java/fixtures/typed_decisions.json
new file mode 100644
index 00000000..6e8a202d
--- /dev/null
+++ b/laya-java/fixtures/typed_decisions.json
@@ -0,0 +1,2868 @@
+{
+ "checkpoint": "typed-decisions",
+ "graph": "laya.onnx",
+ "config": {
+ "model_name": "laya-typed-decisions",
+ "encoder": "answerdotai/ModernBERT-large",
+ "max_len": 1024,
+ "head_max_len": 256,
+ "temperature": [
+ 1.0148024559020996,
+ 1.0374259948730469,
+ 1.0575125217437744
+ ],
+ "temperature_by_options": {
+ "choice:3-5": 1.7601518630981445,
+ "choice:6-10": 1.0000158548355103,
+ "score:3-5": 1.2514300346374512,
+ "noul:2": 1.983399510383606,
+ "choice:11+": 0.5,
+ "choice:2": 1.9063563346862793
+ },
+ "temperature_raw": [
+ 1.0148024559020996,
+ 1.0374259948730469,
+ 1.0575125217437744
+ ],
+ "temperature_by_options_raw": {
+ "choice:3-5": 1.7601518630981445,
+ "choice:6-10": 1.0000158548355103,
+ "score:3-5": 1.2514300346374512,
+ "noul:2": 1.983399510383606,
+ "choice:11+": 0.10058280825614929,
+ "choice:2": 1.9063563346862793
+ }
+ },
+ "workflows": {
+ "agent_trace_observability": [
+ "action",
+ "needs_review",
+ "outcome",
+ "risk",
+ "urgency"
+ ],
+ "customer_service": [
+ "action",
+ "category",
+ "churn_risk",
+ "needs_human",
+ "urgency"
+ ],
+ "invoice_processing": [
+ "discrepancy_severity",
+ "disposition",
+ "duplicate",
+ "matches_order",
+ "urgency"
+ ],
+ "security_incidents": [
+ "credential_compromise",
+ "disposition",
+ "severity",
+ "true_positive",
+ "urgency"
+ ]
+ },
+ "single": {
+ "invoice-variance": {
+ "state": {
+ "invoice": {
+ "id": "INV-2026-8842",
+ "vendor": "Northwind Castings",
+ "currency": "USD",
+ "total_usd": 300020.0,
+ "lines": [
+ {
+ "sku": "SKU-940",
+ "qty": 1000,
+ "unit_usd": 300.02,
+ "total_usd": 300020.0
+ }
+ ]
+ },
+ "purchase_order": {
+ "id": "PO-3392",
+ "total_usd": 214300.0,
+ "lines": [
+ {
+ "qty": 1000,
+ "unit_usd": 214.3
+ }
+ ],
+ "freight_terms": "freight prepaid by vendor, not separately billable"
+ },
+ "delivery": {
+ "date": "2026-03-18",
+ "received_qty": 1000,
+ "condition": "accepted with exceptions"
+ },
+ "payment": {
+ "terms": "net 45",
+ "status": "overdue",
+ "days_until_due": -3
+ },
+ "vendor_history": {
+ "invoices_12m": 40,
+ "disputes_12m": 0,
+ "prior_invoice_ids": [
+ "INV-2026-6633",
+ "INV-2026-9563",
+ "INV-2026-7939"
+ ]
+ }
+ },
+ "lang": null,
+ "workflow": "invoice_processing",
+ "questions": {
+ "disposition": {
+ "type": "choice",
+ "instructions": "How should this vendor invoice be dispositioned?",
+ "criteria": {
+ "approve": "Matches the order and delivery; approve for payment.",
+ "hold": "Something needs confirming before payment; hold pending clarification.",
+ "manual_review": "A human in finance must review the discrepancy.",
+ "reject": "Should not be paid: duplicate, unauthorised or materially wrong."
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "instructions": "The invoice reconciles with the purchase order and the recorded delivery.",
+ "criteria": {
+ "false": "There is a discrepancy against the order or the delivery.",
+ "true": "Line items, quantities and amounts reconcile."
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "instructions": "This invoice appears to duplicate an invoice already submitted."
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "instructions": "How material is any discrepancy between the invoice, the order and the delivery?",
+ "criteria": [
+ "None: everything reconciles.",
+ "Trivial: rounding or a cosmetic difference.",
+ "Moderate: a real difference worth confirming.",
+ "Material: a large or unexplained difference."
+ ]
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How time-sensitive is processing this invoice?",
+ "criteria": [
+ "No time pressure; can wait indefinitely.",
+ "Routine; handle within the normal queue.",
+ "Elevated; should be handled within the same week.",
+ "Critical; requires action within the same day."
+ ]
+ }
+ },
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.0908,
+ "hold": 0.3301,
+ "manual_review": 0.4534,
+ "reject": 0.1257
+ },
+ "confidence": 0.1322,
+ "answer_confidence": 0.4534,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.2063,
+ "confidence": 0.7937,
+ "answer_confidence": 0.7937,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.1399,
+ "confidence": 0.8601,
+ "answer_confidence": 0.8601,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 2.3799,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.0405,
+ "1": 0.0602,
+ "2": 0.378,
+ "3": 0.5212
+ },
+ "confidence": 0.2739,
+ "answer_confidence": 0.5212,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 2.1864,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0226,
+ "1": 0.1418,
+ "2": 0.4621,
+ "3": 0.3735
+ },
+ "confidence": 0.2157,
+ "answer_confidence": 0.4621,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 1666,
+ "output_tokens": 0,
+ "state_tokens": 274,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "invoice-clean": {
+ "state": {
+ "invoice": {
+ "id": "INV-2026-9107",
+ "vendor": "Cedar Freight",
+ "currency": "USD",
+ "total_usd": 18450.0,
+ "lines": [
+ {
+ "sku": "SKU-221",
+ "qty": 300,
+ "unit_usd": 61.5,
+ "total_usd": 18450.0
+ }
+ ]
+ },
+ "purchase_order": {
+ "id": "PO-3477",
+ "total_usd": 18450.0,
+ "lines": [
+ {
+ "qty": 300,
+ "unit_usd": 61.5
+ }
+ ],
+ "freight_terms": "freight prepaid by vendor, not separately billable"
+ },
+ "delivery": {
+ "date": "2026-03-20",
+ "received_qty": 300,
+ "condition": "accepted in full"
+ },
+ "payment": {
+ "terms": "net 30",
+ "status": "current",
+ "days_until_due": 18
+ },
+ "vendor_history": {
+ "invoices_12m": 12,
+ "disputes_12m": 0,
+ "prior_invoice_ids": [
+ "INV-2026-7781"
+ ]
+ }
+ },
+ "lang": null,
+ "workflow": "invoice_processing",
+ "questions": {
+ "disposition": {
+ "type": "choice",
+ "instructions": "How should this vendor invoice be dispositioned?",
+ "criteria": {
+ "approve": "Matches the order and delivery; approve for payment.",
+ "hold": "Something needs confirming before payment; hold pending clarification.",
+ "manual_review": "A human in finance must review the discrepancy.",
+ "reject": "Should not be paid: duplicate, unauthorised or materially wrong."
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "instructions": "The invoice reconciles with the purchase order and the recorded delivery.",
+ "criteria": {
+ "false": "There is a discrepancy against the order or the delivery.",
+ "true": "Line items, quantities and amounts reconcile."
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "instructions": "This invoice appears to duplicate an invoice already submitted."
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "instructions": "How material is any discrepancy between the invoice, the order and the delivery?",
+ "criteria": [
+ "None: everything reconciles.",
+ "Trivial: rounding or a cosmetic difference.",
+ "Moderate: a real difference worth confirming.",
+ "Material: a large or unexplained difference."
+ ]
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How time-sensitive is processing this invoice?",
+ "criteria": [
+ "No time pressure; can wait indefinitely.",
+ "Routine; handle within the normal queue.",
+ "Elevated; should be handled within the same week.",
+ "Critical; requires action within the same day."
+ ]
+ }
+ },
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "approve",
+ "probabilities": {
+ "approve": 0.4076,
+ "hold": 0.2312,
+ "manual_review": 0.2092,
+ "reject": 0.152
+ },
+ "confidence": 0.0492,
+ "answer_confidence": 0.4076,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.6562,
+ "confidence": 0.6562,
+ "answer_confidence": 0.6562,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.1452,
+ "confidence": 0.8548,
+ "answer_confidence": 0.8548,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.0056,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.4979,
+ "1": 0.1355,
+ "2": 0.2296,
+ "3": 0.1369
+ },
+ "confidence": 0.1141,
+ "answer_confidence": 0.4979,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5989,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0413,
+ "1": 0.441,
+ "2": 0.3952,
+ "3": 0.1225
+ },
+ "confidence": 0.1945,
+ "answer_confidence": 0.441,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 1556,
+ "output_tokens": 0,
+ "state_tokens": 252,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "customer-service": {
+ "state": "We were billed twice for March. Third time this year -- refund it today or we are moving to a competitor.",
+ "lang": null,
+ "workflow": "customer_service",
+ "questions": {
+ "action": {
+ "type": "choice",
+ "instructions": "What should the support team do next?",
+ "criteria": {
+ "refund": "Return the money and close the ticket.",
+ "replace": "Ship a replacement unit.",
+ "explain": "Answer the question; nothing to send.",
+ "escalate": "Hand to a senior agent."
+ }
+ },
+ "category": {
+ "type": "choice",
+ "instructions": "Which queue does this ticket belong in?",
+ "criteria": {
+ "billing": "Charges, invoices and refunds.",
+ "shipping": "Delivery, tracking and damage in transit.",
+ "account": "Login, access and account settings."
+ }
+ },
+ "churn_risk": {
+ "type": "score",
+ "instructions": "How likely is this customer to leave over this?",
+ "criteria": [
+ "Not at risk; routine contact.",
+ "Mildly annoyed.",
+ "Considering alternatives.",
+ "Explicitly threatening to cancel."
+ ]
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This ticket needs a human agent rather than an automated reply."
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How quickly must this be answered?",
+ "criteria": [
+ "No time pressure.",
+ "Within the normal queue.",
+ "Same week.",
+ "Same day."
+ ]
+ }
+ },
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "action": {
+ "type": "choice",
+ "choice": "refund",
+ "probabilities": {
+ "refund": 0.6664,
+ "replace": 0.0648,
+ "explain": 0.0534,
+ "escalate": 0.2154
+ },
+ "confidence": 0.3256,
+ "answer_confidence": 0.6664,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "category": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8789,
+ "shipping": 0.0432,
+ "account": 0.0779
+ },
+ "confidence": 0.5922,
+ "answer_confidence": 0.8789,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "churn_risk": {
+ "type": "score",
+ "score": 2.4863,
+ "legend": {
+ "0": "Not at risk; routine contact.",
+ "1": "Mildly annoyed.",
+ "2": "Considering alternatives.",
+ "3": "Explicitly threatening to cancel."
+ },
+ "probabilities": {
+ "0": 0.0101,
+ "1": 0.0936,
+ "2": 0.2962,
+ "3": 0.6001
+ },
+ "confidence": 0.3256,
+ "answer_confidence": 0.6001,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "needs_human": {
+ "type": "noul",
+ "noul": 0.6657,
+ "confidence": 0.6657,
+ "answer_confidence": 0.6657,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 2.6585,
+ "legend": {
+ "0": "No time pressure.",
+ "1": "Within the normal queue.",
+ "2": "Same week.",
+ "3": "Same day."
+ },
+ "probabilities": {
+ "0": 0.0112,
+ "1": 0.0364,
+ "2": 0.2353,
+ "3": 0.7172
+ },
+ "confidence": 0.4593,
+ "answer_confidence": 0.7172,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 355,
+ "output_tokens": 0,
+ "state_tokens": 23,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "security-incident": {
+ "state": {
+ "alert": "impossible travel",
+ "user": "[email protected]",
+ "signins": [
+ {
+ "city": "Lagos",
+ "at": "2026-03-18T09:02Z"
+ },
+ {
+ "city": "Vilnius",
+ "at": "2026-03-18T09:41Z"
+ }
+ ],
+ "mfa": "satisfied by push, approved on the second prompt",
+ "prior_alerts_30d": 0
+ },
+ "lang": null,
+ "workflow": "security_incidents",
+ "questions": {
+ "credential_compromise": {
+ "type": "noul",
+ "instructions": "Credentials belonging to this account were exposed or reused.",
+ "criteria": {
+ "false": "No sign that a secret left the organisation.",
+ "true": "A password, token or key is in the wrong hands."
+ }
+ },
+ "disposition": {
+ "type": "choice",
+ "instructions": "How should this alert be dispositioned?",
+ "criteria": {
+ "close": "Benign; close with no action.",
+ "monitor": "Watch for a repeat before acting.",
+ "contain": "Isolate the host or revoke the credential now.",
+ "escalate": "Wake an incident responder."
+ }
+ },
+ "severity": {
+ "type": "score",
+ "instructions": "How severe is the impact if this is real?",
+ "criteria": [
+ "Informational.",
+ "Low: one account, no data.",
+ "High: production data or lateral movement.",
+ "Critical: active, spreading, or customer data."
+ ]
+ },
+ "true_positive": {
+ "type": "noul",
+ "instructions": "This alert describes genuine malicious activity rather than noise."
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How quickly must someone respond?",
+ "criteria": [
+ "No time pressure.",
+ "Next business day.",
+ "Within the hour.",
+ "Immediately."
+ ]
+ }
+ },
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "credential_compromise": {
+ "type": "noul",
+ "noul": 0.2801,
+ "confidence": 0.7199,
+ "answer_confidence": 0.7199,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "disposition": {
+ "type": "choice",
+ "choice": "close",
+ "probabilities": {
+ "close": 0.3521,
+ "monitor": 0.2603,
+ "contain": 0.1785,
+ "escalate": 0.2091
+ },
+ "confidence": 0.0242,
+ "answer_confidence": 0.3521,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "severity": {
+ "type": "score",
+ "score": 1.1706,
+ "legend": {
+ "0": "Informational.",
+ "1": "Low: one account, no data.",
+ "2": "High: production data or lateral movement.",
+ "3": "Critical: active, spreading, or customer data."
+ },
+ "probabilities": {
+ "0": 0.3117,
+ "1": 0.3074,
+ "2": 0.2794,
+ "3": 0.1015
+ },
+ "confidence": 0.0518,
+ "answer_confidence": 0.3117,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "true_positive": {
+ "type": "noul",
+ "noul": 0.1774,
+ "confidence": 0.8226,
+ "answer_confidence": 0.8226,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.7277,
+ "legend": {
+ "0": "No time pressure.",
+ "1": "Next business day.",
+ "2": "Within the hour.",
+ "3": "Immediately."
+ },
+ "probabilities": {
+ "0": 0.1175,
+ "1": 0.2869,
+ "2": 0.3459,
+ "3": 0.2496
+ },
+ "confidence": 0.0453,
+ "answer_confidence": 0.3459,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 745,
+ "output_tokens": 0,
+ "state_tokens": 99,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "agent-trace": {
+ "state": {
+ "run": "agt-7741",
+ "tool_calls": 14,
+ "wrote": [
+ "s3://prod-reports/q1.csv"
+ ],
+ "errors": [
+ "RateLimit on call 9, retried"
+ ],
+ "final": "uploaded the report, but the row count is 0"
+ },
+ "lang": null,
+ "workflow": "agent_trace_observability",
+ "questions": {
+ "action": {
+ "type": "choice",
+ "instructions": "What should be done with this agent run?",
+ "criteria": {
+ "accept": "The run did what was asked; accept it.",
+ "retry": "Transient failure; run it again.",
+ "rollback": "Undo what it changed.",
+ "escalate": "A person must look at this."
+ }
+ },
+ "needs_review": {
+ "type": "noul",
+ "instructions": "A human should read this trace before the result is used."
+ },
+ "outcome": {
+ "type": "choice",
+ "instructions": "How did the run end?",
+ "criteria": {
+ "success": "Completed the task.",
+ "partial": "Completed some of it.",
+ "failure": "Did not complete the task."
+ }
+ },
+ "risk": {
+ "type": "score",
+ "instructions": "How much damage could this run's actions have caused?",
+ "criteria": [
+ "Read-only; no effect.",
+ "Reversible writes.",
+ "Irreversible writes to internal systems.",
+ "Irreversible, customer-visible, or spending money."
+ ]
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How quickly must this be looked at?",
+ "criteria": [
+ "No time pressure.",
+ "Next business day.",
+ "Within the hour.",
+ "Immediately."
+ ]
+ }
+ },
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "action": {
+ "type": "choice",
+ "choice": "retry",
+ "probabilities": {
+ "accept": 0.1787,
+ "retry": 0.367,
+ "rollback": 0.1785,
+ "escalate": 0.2758
+ },
+ "confidence": 0.0345,
+ "answer_confidence": 0.367,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "needs_review": {
+ "type": "noul",
+ "noul": 0.3861,
+ "confidence": 0.6139,
+ "answer_confidence": 0.6139,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "outcome": {
+ "type": "choice",
+ "choice": "failure",
+ "probabilities": {
+ "success": 0.1267,
+ "partial": 0.3769,
+ "failure": 0.4964
+ },
+ "confidence": 0.1106,
+ "answer_confidence": 0.4964,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "risk": {
+ "type": "score",
+ "score": 1.4808,
+ "legend": {
+ "0": "Read-only; no effect.",
+ "1": "Reversible writes.",
+ "2": "Irreversible writes to internal systems.",
+ "3": "Irreversible, customer-visible, or spending money."
+ },
+ "probabilities": {
+ "0": 0.1546,
+ "1": 0.3994,
+ "2": 0.2565,
+ "3": 0.1894
+ },
+ "confidence": 0.0483,
+ "answer_confidence": 0.3994,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.8262,
+ "legend": {
+ "0": "No time pressure.",
+ "1": "Next business day.",
+ "2": "Within the hour.",
+ "3": "Immediately."
+ },
+ "probabilities": {
+ "0": 0.0736,
+ "1": 0.3121,
+ "2": 0.3286,
+ "3": 0.2856
+ },
+ "confidence": 0.0773,
+ "answer_confidence": 0.3286,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 558,
+ "output_tokens": 0,
+ "state_tokens": 62,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "many-options": {
+ "state": "The invoice totals 300,020 USD against a 214,300 USD order.",
+ "lang": null,
+ "workflow": null,
+ "questions": {
+ "band": {
+ "type": "choice",
+ "instructions": "Which spend band does this invoice fall in?",
+ "criteria": {
+ "band00": "tier number 0",
+ "band01": "tier number 1",
+ "band02": "tier number 2",
+ "band03": "tier number 3",
+ "band04": "tier number 4",
+ "band05": "tier number 5",
+ "band06": "tier number 6",
+ "band07": "tier number 7",
+ "band08": "tier number 8",
+ "band09": "tier number 9",
+ "band10": "tier number 10",
+ "band11": "tier number 11",
+ "band12": "tier number 12",
+ "band13": "tier number 13"
+ }
+ }
+ },
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "band": {
+ "type": "choice",
+ "choice": "band01",
+ "probabilities": {
+ "band00": 0.1393,
+ "band01": 0.3035,
+ "band02": 0.1856,
+ "band03": 0.0873,
+ "band04": 0.0365,
+ "band05": 0.0311,
+ "band06": 0.0313,
+ "band07": 0.0304,
+ "band08": 0.0349,
+ "band09": 0.043,
+ "band10": 0.0163,
+ "band11": 0.0132,
+ "band12": 0.0218,
+ "band13": 0.0257
+ },
+ "confidence": 0.1817,
+ "answer_confidence": 0.3035,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 130,
+ "output_tokens": 0,
+ "state_tokens": 16,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ "over-budget": {
+ "state": "line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. line item number one, unit price 300.02 against a purchase order at 214.30, delivery accepted with exceptions. ",
+ "lang": null,
+ "workflow": "invoice_processing",
+ "questions": {
+ "disposition": {
+ "type": "choice",
+ "instructions": "How should this vendor invoice be dispositioned?",
+ "criteria": {
+ "approve": "Matches the order and delivery; approve for payment.",
+ "hold": "Something needs confirming before payment; hold pending clarification.",
+ "manual_review": "A human in finance must review the discrepancy.",
+ "reject": "Should not be paid: duplicate, unauthorised or materially wrong."
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "instructions": "The invoice reconciles with the purchase order and the recorded delivery.",
+ "criteria": {
+ "false": "There is a discrepancy against the order or the delivery.",
+ "true": "Line items, quantities and amounts reconcile."
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "instructions": "This invoice appears to duplicate an invoice already submitted."
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "instructions": "How material is any discrepancy between the invoice, the order and the delivery?",
+ "criteria": [
+ "None: everything reconciles.",
+ "Trivial: rounding or a cosmetic difference.",
+ "Moderate: a real difference worth confirming.",
+ "Material: a large or unexplained difference."
+ ]
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How time-sensitive is processing this invoice?",
+ "criteria": [
+ "No time pressure; can wait indefinitely.",
+ "Routine; handle within the normal queue.",
+ "Elevated; should be handled within the same week.",
+ "Critical; requires action within the same day."
+ ]
+ }
+ },
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.2397,
+ "hold": 0.2893,
+ "manual_review": 0.3046,
+ "reject": 0.1664
+ },
+ "confidence": 0.0177,
+ "answer_confidence": 0.3046,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.4998,
+ "confidence": 0.5002,
+ "answer_confidence": 0.5002,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.2023,
+ "confidence": 0.7977,
+ "answer_confidence": 0.7977,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.7091,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.2096,
+ "1": 0.1382,
+ "2": 0.3856,
+ "3": 0.2666
+ },
+ "confidence": 0.0472,
+ "answer_confidence": 0.3856,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.4485,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0834,
+ "1": 0.4863,
+ "2": 0.3288,
+ "3": 0.1015
+ },
+ "confidence": 0.1663,
+ "answer_confidence": 0.4863,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 5120,
+ "output_tokens": 0,
+ "state_tokens": 2161,
+ "state_tokens_dropped": 1211,
+ "truncated": true,
+ "truncated_questions": [
+ "disposition",
+ "matches_order",
+ "duplicate",
+ "discrepancy_severity",
+ "urgency"
+ ]
+ }
+ },
+ "off-workflow": {
+ "state": "We were billed twice for March and want a refund today.",
+ "lang": "en",
+ "workflow": null,
+ "questions": {
+ "department": {
+ "type": "choice",
+ "instructions": "Which team owns this?",
+ "criteria": {
+ "billing": "charges and refunds",
+ "other": "anything else"
+ }
+ }
+ },
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "department": {
+ "type": "choice",
+ "choice": "billing",
+ "probabilities": {
+ "billing": 0.8967,
+ "other": 0.1033
+ },
+ "confidence": 0.5207,
+ "answer_confidence": 0.8967,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 36,
+ "output_tokens": 0,
+ "state_tokens": 12,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ },
+ "batch": [
+ {
+ "batch_size": null,
+ "sort_by_length": false,
+ "states": [
+ {
+ "invoice": {
+ "id": "INV-2026-8842",
+ "vendor": "Northwind Castings",
+ "currency": "USD",
+ "total_usd": 300020.0,
+ "lines": [
+ {
+ "sku": "SKU-940",
+ "qty": 1000,
+ "unit_usd": 300.02,
+ "total_usd": 300020.0
+ }
+ ]
+ },
+ "purchase_order": {
+ "id": "PO-3392",
+ "total_usd": 214300.0,
+ "lines": [
+ {
+ "qty": 1000,
+ "unit_usd": 214.3
+ }
+ ],
+ "freight_terms": "freight prepaid by vendor, not separately billable"
+ },
+ "delivery": {
+ "date": "2026-03-18",
+ "received_qty": 1000,
+ "condition": "accepted with exceptions"
+ },
+ "payment": {
+ "terms": "net 45",
+ "status": "overdue",
+ "days_until_due": -3
+ },
+ "vendor_history": {
+ "invoices_12m": 40,
+ "disputes_12m": 0,
+ "prior_invoice_ids": [
+ "INV-2026-6633",
+ "INV-2026-9563",
+ "INV-2026-7939"
+ ]
+ }
+ },
+ {
+ "invoice": {
+ "id": "INV-2026-9107",
+ "vendor": "Cedar Freight",
+ "currency": "USD",
+ "total_usd": 18450.0,
+ "lines": [
+ {
+ "sku": "SKU-221",
+ "qty": 300,
+ "unit_usd": 61.5,
+ "total_usd": 18450.0
+ }
+ ]
+ },
+ "purchase_order": {
+ "id": "PO-3477",
+ "total_usd": 18450.0,
+ "lines": [
+ {
+ "qty": 300,
+ "unit_usd": 61.5
+ }
+ ],
+ "freight_terms": "freight prepaid by vendor, not separately billable"
+ },
+ "delivery": {
+ "date": "2026-03-20",
+ "received_qty": 300,
+ "condition": "accepted in full"
+ },
+ "payment": {
+ "terms": "net 30",
+ "status": "current",
+ "days_until_due": 18
+ },
+ "vendor_history": {
+ "invoices_12m": 12,
+ "disputes_12m": 0,
+ "prior_invoice_ids": [
+ "INV-2026-7781"
+ ]
+ }
+ },
+ "INV-2026-1004, 980 USD, matches PO-3100 and delivery.",
+ "duplicate of INV-2026-1004",
+ ""
+ ],
+ "questions": {
+ "disposition": {
+ "type": "choice",
+ "instructions": "How should this vendor invoice be dispositioned?",
+ "criteria": {
+ "approve": "Matches the order and delivery; approve for payment.",
+ "hold": "Something needs confirming before payment; hold pending clarification.",
+ "manual_review": "A human in finance must review the discrepancy.",
+ "reject": "Should not be paid: duplicate, unauthorised or materially wrong."
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "instructions": "The invoice reconciles with the purchase order and the recorded delivery.",
+ "criteria": {
+ "false": "There is a discrepancy against the order or the delivery.",
+ "true": "Line items, quantities and amounts reconcile."
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "instructions": "This invoice appears to duplicate an invoice already submitted."
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "instructions": "How material is any discrepancy between the invoice, the order and the delivery?",
+ "criteria": [
+ "None: everything reconciles.",
+ "Trivial: rounding or a cosmetic difference.",
+ "Moderate: a real difference worth confirming.",
+ "Material: a large or unexplained difference."
+ ]
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How time-sensitive is processing this invoice?",
+ "criteria": [
+ "No time pressure; can wait indefinitely.",
+ "Routine; handle within the normal queue.",
+ "Elevated; should be handled within the same week.",
+ "Critical; requires action within the same day."
+ ]
+ }
+ },
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.0908,
+ "hold": 0.3301,
+ "manual_review": 0.4534,
+ "reject": 0.1257
+ },
+ "confidence": 0.1322,
+ "answer_confidence": 0.4534,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.2063,
+ "confidence": 0.7937,
+ "answer_confidence": 0.7937,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.1399,
+ "confidence": 0.8601,
+ "answer_confidence": 0.8601,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 2.3799,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.0405,
+ "1": 0.0602,
+ "2": 0.378,
+ "3": 0.5212
+ },
+ "confidence": 0.2739,
+ "answer_confidence": 0.5212,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 2.1864,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0226,
+ "1": 0.1418,
+ "2": 0.4621,
+ "3": 0.3735
+ },
+ "confidence": 0.2157,
+ "answer_confidence": 0.4621,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 1666,
+ "output_tokens": 0,
+ "state_tokens": 274,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "approve",
+ "probabilities": {
+ "approve": 0.4076,
+ "hold": 0.2312,
+ "manual_review": 0.2092,
+ "reject": 0.152
+ },
+ "confidence": 0.0492,
+ "answer_confidence": 0.4076,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.6562,
+ "confidence": 0.6562,
+ "answer_confidence": 0.6562,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.1452,
+ "confidence": 0.8548,
+ "answer_confidence": 0.8548,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.0056,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.4979,
+ "1": 0.1355,
+ "2": 0.2296,
+ "3": 0.1369
+ },
+ "confidence": 0.1141,
+ "answer_confidence": 0.4979,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5989,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0413,
+ "1": 0.441,
+ "2": 0.3952,
+ "3": 0.1225
+ },
+ "confidence": 0.1945,
+ "answer_confidence": 0.441,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 1556,
+ "output_tokens": 0,
+ "state_tokens": 252,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "approve",
+ "probabilities": {
+ "approve": 0.5173,
+ "hold": 0.1756,
+ "manual_review": 0.1803,
+ "reject": 0.1267
+ },
+ "confidence": 0.122,
+ "answer_confidence": 0.5173,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.7186,
+ "confidence": 0.7186,
+ "answer_confidence": 0.7186,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.2763,
+ "confidence": 0.7237,
+ "answer_confidence": 0.7237,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 0.6375,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.6903,
+ "1": 0.0668,
+ "2": 0.158,
+ "3": 0.0849
+ },
+ "confidence": 0.3236,
+ "answer_confidence": 0.6903,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5688,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0407,
+ "1": 0.47,
+ "2": 0.369,
+ "3": 0.1203
+ },
+ "confidence": 0.2008,
+ "answer_confidence": 0.47,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 396,
+ "output_tokens": 0,
+ "state_tokens": 20,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.1746,
+ "hold": 0.2887,
+ "manual_review": 0.3221,
+ "reject": 0.2145
+ },
+ "confidence": 0.02,
+ "answer_confidence": 0.3221,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.6206,
+ "confidence": 0.6206,
+ "answer_confidence": 0.6206,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.4192,
+ "confidence": 0.5808,
+ "answer_confidence": 0.5808,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.823,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.1938,
+ "1": 0.1166,
+ "2": 0.3623,
+ "3": 0.3273
+ },
+ "confidence": 0.0608,
+ "answer_confidence": 0.3623,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.603,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0369,
+ "1": 0.4401,
+ "2": 0.4062,
+ "3": 0.1169
+ },
+ "confidence": 0.2067,
+ "answer_confidence": 0.4401,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 346,
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.2066,
+ "hold": 0.3032,
+ "manual_review": 0.335,
+ "reject": 0.1552
+ },
+ "confidence": 0.0311,
+ "answer_confidence": 0.335,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.5773,
+ "confidence": 0.5773,
+ "answer_confidence": 0.5773,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.15,
+ "confidence": 0.85,
+ "answer_confidence": 0.85,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.5307,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.3219,
+ "1": 0.1028,
+ "2": 0.2982,
+ "3": 0.2772
+ },
+ "confidence": 0.0513,
+ "answer_confidence": 0.3219,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5328,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0409,
+ "1": 0.4738,
+ "2": 0.397,
+ "3": 0.0883
+ },
+ "confidence": 0.2313,
+ "answer_confidence": 0.4738,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 296,
+ "output_tokens": 0,
+ "state_tokens": 0,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ]
+ },
+ {
+ "batch_size": 2,
+ "sort_by_length": false,
+ "states": [
+ {
+ "invoice": {
+ "id": "INV-2026-8842",
+ "vendor": "Northwind Castings",
+ "currency": "USD",
+ "total_usd": 300020.0,
+ "lines": [
+ {
+ "sku": "SKU-940",
+ "qty": 1000,
+ "unit_usd": 300.02,
+ "total_usd": 300020.0
+ }
+ ]
+ },
+ "purchase_order": {
+ "id": "PO-3392",
+ "total_usd": 214300.0,
+ "lines": [
+ {
+ "qty": 1000,
+ "unit_usd": 214.3
+ }
+ ],
+ "freight_terms": "freight prepaid by vendor, not separately billable"
+ },
+ "delivery": {
+ "date": "2026-03-18",
+ "received_qty": 1000,
+ "condition": "accepted with exceptions"
+ },
+ "payment": {
+ "terms": "net 45",
+ "status": "overdue",
+ "days_until_due": -3
+ },
+ "vendor_history": {
+ "invoices_12m": 40,
+ "disputes_12m": 0,
+ "prior_invoice_ids": [
+ "INV-2026-6633",
+ "INV-2026-9563",
+ "INV-2026-7939"
+ ]
+ }
+ },
+ {
+ "invoice": {
+ "id": "INV-2026-9107",
+ "vendor": "Cedar Freight",
+ "currency": "USD",
+ "total_usd": 18450.0,
+ "lines": [
+ {
+ "sku": "SKU-221",
+ "qty": 300,
+ "unit_usd": 61.5,
+ "total_usd": 18450.0
+ }
+ ]
+ },
+ "purchase_order": {
+ "id": "PO-3477",
+ "total_usd": 18450.0,
+ "lines": [
+ {
+ "qty": 300,
+ "unit_usd": 61.5
+ }
+ ],
+ "freight_terms": "freight prepaid by vendor, not separately billable"
+ },
+ "delivery": {
+ "date": "2026-03-20",
+ "received_qty": 300,
+ "condition": "accepted in full"
+ },
+ "payment": {
+ "terms": "net 30",
+ "status": "current",
+ "days_until_due": 18
+ },
+ "vendor_history": {
+ "invoices_12m": 12,
+ "disputes_12m": 0,
+ "prior_invoice_ids": [
+ "INV-2026-7781"
+ ]
+ }
+ },
+ "INV-2026-1004, 980 USD, matches PO-3100 and delivery.",
+ "duplicate of INV-2026-1004",
+ ""
+ ],
+ "questions": {
+ "disposition": {
+ "type": "choice",
+ "instructions": "How should this vendor invoice be dispositioned?",
+ "criteria": {
+ "approve": "Matches the order and delivery; approve for payment.",
+ "hold": "Something needs confirming before payment; hold pending clarification.",
+ "manual_review": "A human in finance must review the discrepancy.",
+ "reject": "Should not be paid: duplicate, unauthorised or materially wrong."
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "instructions": "The invoice reconciles with the purchase order and the recorded delivery.",
+ "criteria": {
+ "false": "There is a discrepancy against the order or the delivery.",
+ "true": "Line items, quantities and amounts reconcile."
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "instructions": "This invoice appears to duplicate an invoice already submitted."
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "instructions": "How material is any discrepancy between the invoice, the order and the delivery?",
+ "criteria": [
+ "None: everything reconciles.",
+ "Trivial: rounding or a cosmetic difference.",
+ "Moderate: a real difference worth confirming.",
+ "Material: a large or unexplained difference."
+ ]
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How time-sensitive is processing this invoice?",
+ "criteria": [
+ "No time pressure; can wait indefinitely.",
+ "Routine; handle within the normal queue.",
+ "Elevated; should be handled within the same week.",
+ "Critical; requires action within the same day."
+ ]
+ }
+ },
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.0908,
+ "hold": 0.3301,
+ "manual_review": 0.4534,
+ "reject": 0.1257
+ },
+ "confidence": 0.1322,
+ "answer_confidence": 0.4534,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.2063,
+ "confidence": 0.7937,
+ "answer_confidence": 0.7937,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.1399,
+ "confidence": 0.8601,
+ "answer_confidence": 0.8601,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 2.3799,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.0405,
+ "1": 0.0602,
+ "2": 0.378,
+ "3": 0.5212
+ },
+ "confidence": 0.2739,
+ "answer_confidence": 0.5212,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 2.1864,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0226,
+ "1": 0.1418,
+ "2": 0.4621,
+ "3": 0.3735
+ },
+ "confidence": 0.2157,
+ "answer_confidence": 0.4621,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 1666,
+ "output_tokens": 0,
+ "state_tokens": 274,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "approve",
+ "probabilities": {
+ "approve": 0.4076,
+ "hold": 0.2312,
+ "manual_review": 0.2092,
+ "reject": 0.152
+ },
+ "confidence": 0.0492,
+ "answer_confidence": 0.4076,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.6562,
+ "confidence": 0.6562,
+ "answer_confidence": 0.6562,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.1452,
+ "confidence": 0.8548,
+ "answer_confidence": 0.8548,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.0056,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.4979,
+ "1": 0.1355,
+ "2": 0.2296,
+ "3": 0.1369
+ },
+ "confidence": 0.1141,
+ "answer_confidence": 0.4979,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5989,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0413,
+ "1": 0.441,
+ "2": 0.3952,
+ "3": 0.1225
+ },
+ "confidence": 0.1945,
+ "answer_confidence": 0.441,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 1556,
+ "output_tokens": 0,
+ "state_tokens": 252,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "approve",
+ "probabilities": {
+ "approve": 0.5173,
+ "hold": 0.1756,
+ "manual_review": 0.1803,
+ "reject": 0.1267
+ },
+ "confidence": 0.122,
+ "answer_confidence": 0.5173,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.7186,
+ "confidence": 0.7186,
+ "answer_confidence": 0.7186,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.2763,
+ "confidence": 0.7237,
+ "answer_confidence": 0.7237,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 0.6375,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.6903,
+ "1": 0.0668,
+ "2": 0.158,
+ "3": 0.0849
+ },
+ "confidence": 0.3236,
+ "answer_confidence": 0.6903,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5688,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0407,
+ "1": 0.47,
+ "2": 0.369,
+ "3": 0.1203
+ },
+ "confidence": 0.2008,
+ "answer_confidence": 0.47,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 396,
+ "output_tokens": 0,
+ "state_tokens": 20,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.1746,
+ "hold": 0.2887,
+ "manual_review": 0.3221,
+ "reject": 0.2145
+ },
+ "confidence": 0.02,
+ "answer_confidence": 0.3221,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.6206,
+ "confidence": 0.6206,
+ "answer_confidence": 0.6206,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.4192,
+ "confidence": 0.5808,
+ "answer_confidence": 0.5808,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.823,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.1938,
+ "1": 0.1166,
+ "2": 0.3623,
+ "3": 0.3273
+ },
+ "confidence": 0.0608,
+ "answer_confidence": 0.3623,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.603,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0369,
+ "1": 0.4401,
+ "2": 0.4062,
+ "3": 0.1169
+ },
+ "confidence": 0.2067,
+ "answer_confidence": 0.4401,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 346,
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.2066,
+ "hold": 0.3032,
+ "manual_review": 0.335,
+ "reject": 0.1552
+ },
+ "confidence": 0.0311,
+ "answer_confidence": 0.335,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.5773,
+ "confidence": 0.5773,
+ "answer_confidence": 0.5773,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.15,
+ "confidence": 0.85,
+ "answer_confidence": 0.85,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.5307,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.3219,
+ "1": 0.1028,
+ "2": 0.2982,
+ "3": 0.2772
+ },
+ "confidence": 0.0513,
+ "answer_confidence": 0.3219,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5328,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0409,
+ "1": 0.4738,
+ "2": 0.397,
+ "3": 0.0883
+ },
+ "confidence": 0.2313,
+ "answer_confidence": 0.4738,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 296,
+ "output_tokens": 0,
+ "state_tokens": 0,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ]
+ },
+ {
+ "batch_size": 3,
+ "sort_by_length": true,
+ "states": [
+ {
+ "invoice": {
+ "id": "INV-2026-8842",
+ "vendor": "Northwind Castings",
+ "currency": "USD",
+ "total_usd": 300020.0,
+ "lines": [
+ {
+ "sku": "SKU-940",
+ "qty": 1000,
+ "unit_usd": 300.02,
+ "total_usd": 300020.0
+ }
+ ]
+ },
+ "purchase_order": {
+ "id": "PO-3392",
+ "total_usd": 214300.0,
+ "lines": [
+ {
+ "qty": 1000,
+ "unit_usd": 214.3
+ }
+ ],
+ "freight_terms": "freight prepaid by vendor, not separately billable"
+ },
+ "delivery": {
+ "date": "2026-03-18",
+ "received_qty": 1000,
+ "condition": "accepted with exceptions"
+ },
+ "payment": {
+ "terms": "net 45",
+ "status": "overdue",
+ "days_until_due": -3
+ },
+ "vendor_history": {
+ "invoices_12m": 40,
+ "disputes_12m": 0,
+ "prior_invoice_ids": [
+ "INV-2026-6633",
+ "INV-2026-9563",
+ "INV-2026-7939"
+ ]
+ }
+ },
+ {
+ "invoice": {
+ "id": "INV-2026-9107",
+ "vendor": "Cedar Freight",
+ "currency": "USD",
+ "total_usd": 18450.0,
+ "lines": [
+ {
+ "sku": "SKU-221",
+ "qty": 300,
+ "unit_usd": 61.5,
+ "total_usd": 18450.0
+ }
+ ]
+ },
+ "purchase_order": {
+ "id": "PO-3477",
+ "total_usd": 18450.0,
+ "lines": [
+ {
+ "qty": 300,
+ "unit_usd": 61.5
+ }
+ ],
+ "freight_terms": "freight prepaid by vendor, not separately billable"
+ },
+ "delivery": {
+ "date": "2026-03-20",
+ "received_qty": 300,
+ "condition": "accepted in full"
+ },
+ "payment": {
+ "terms": "net 30",
+ "status": "current",
+ "days_until_due": 18
+ },
+ "vendor_history": {
+ "invoices_12m": 12,
+ "disputes_12m": 0,
+ "prior_invoice_ids": [
+ "INV-2026-7781"
+ ]
+ }
+ },
+ "INV-2026-1004, 980 USD, matches PO-3100 and delivery.",
+ "duplicate of INV-2026-1004",
+ ""
+ ],
+ "questions": {
+ "disposition": {
+ "type": "choice",
+ "instructions": "How should this vendor invoice be dispositioned?",
+ "criteria": {
+ "approve": "Matches the order and delivery; approve for payment.",
+ "hold": "Something needs confirming before payment; hold pending clarification.",
+ "manual_review": "A human in finance must review the discrepancy.",
+ "reject": "Should not be paid: duplicate, unauthorised or materially wrong."
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "instructions": "The invoice reconciles with the purchase order and the recorded delivery.",
+ "criteria": {
+ "false": "There is a discrepancy against the order or the delivery.",
+ "true": "Line items, quantities and amounts reconcile."
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "instructions": "This invoice appears to duplicate an invoice already submitted."
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "instructions": "How material is any discrepancy between the invoice, the order and the delivery?",
+ "criteria": [
+ "None: everything reconciles.",
+ "Trivial: rounding or a cosmetic difference.",
+ "Moderate: a real difference worth confirming.",
+ "Material: a large or unexplained difference."
+ ]
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How time-sensitive is processing this invoice?",
+ "criteria": [
+ "No time pressure; can wait indefinitely.",
+ "Routine; handle within the normal queue.",
+ "Elevated; should be handled within the same week.",
+ "Critical; requires action within the same day."
+ ]
+ }
+ },
+ "results": [
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.0908,
+ "hold": 0.3301,
+ "manual_review": 0.4534,
+ "reject": 0.1257
+ },
+ "confidence": 0.1322,
+ "answer_confidence": 0.4534,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.2063,
+ "confidence": 0.7937,
+ "answer_confidence": 0.7937,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.1399,
+ "confidence": 0.8601,
+ "answer_confidence": 0.8601,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 2.3799,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.0405,
+ "1": 0.0602,
+ "2": 0.378,
+ "3": 0.5212
+ },
+ "confidence": 0.2739,
+ "answer_confidence": 0.5212,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 2.1864,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0226,
+ "1": 0.1418,
+ "2": 0.4621,
+ "3": 0.3735
+ },
+ "confidence": 0.2157,
+ "answer_confidence": 0.4621,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 1666,
+ "output_tokens": 0,
+ "state_tokens": 274,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "approve",
+ "probabilities": {
+ "approve": 0.4076,
+ "hold": 0.2312,
+ "manual_review": 0.2092,
+ "reject": 0.152
+ },
+ "confidence": 0.0492,
+ "answer_confidence": 0.4076,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.6562,
+ "confidence": 0.6562,
+ "answer_confidence": 0.6562,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.1452,
+ "confidence": 0.8548,
+ "answer_confidence": 0.8548,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.0056,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.4979,
+ "1": 0.1355,
+ "2": 0.2296,
+ "3": 0.1369
+ },
+ "confidence": 0.1141,
+ "answer_confidence": 0.4979,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5989,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0413,
+ "1": 0.441,
+ "2": 0.3952,
+ "3": 0.1225
+ },
+ "confidence": 0.1945,
+ "answer_confidence": 0.441,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 1556,
+ "output_tokens": 0,
+ "state_tokens": 252,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "approve",
+ "probabilities": {
+ "approve": 0.5173,
+ "hold": 0.1756,
+ "manual_review": 0.1803,
+ "reject": 0.1267
+ },
+ "confidence": 0.122,
+ "answer_confidence": 0.5173,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.7186,
+ "confidence": 0.7186,
+ "answer_confidence": 0.7186,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.2763,
+ "confidence": 0.7237,
+ "answer_confidence": 0.7237,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 0.6375,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.6903,
+ "1": 0.0668,
+ "2": 0.158,
+ "3": 0.0849
+ },
+ "confidence": 0.3236,
+ "answer_confidence": 0.6903,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5688,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0407,
+ "1": 0.47,
+ "2": 0.369,
+ "3": 0.1203
+ },
+ "confidence": 0.2008,
+ "answer_confidence": 0.47,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 396,
+ "output_tokens": 0,
+ "state_tokens": 20,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.1746,
+ "hold": 0.2887,
+ "manual_review": 0.3221,
+ "reject": 0.2145
+ },
+ "confidence": 0.02,
+ "answer_confidence": 0.3221,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.6206,
+ "confidence": 0.6206,
+ "answer_confidence": 0.6206,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.4192,
+ "confidence": 0.5808,
+ "answer_confidence": 0.5808,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.823,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.1938,
+ "1": 0.1166,
+ "2": 0.3623,
+ "3": 0.3273
+ },
+ "confidence": 0.0608,
+ "answer_confidence": 0.3623,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.603,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0369,
+ "1": 0.4401,
+ "2": 0.4062,
+ "3": 0.1169
+ },
+ "confidence": 0.2067,
+ "answer_confidence": 0.4401,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 346,
+ "output_tokens": 0,
+ "state_tokens": 10,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "model": "laya-rl-agent-onnx",
+ "answers": {
+ "disposition": {
+ "type": "choice",
+ "choice": "manual_review",
+ "probabilities": {
+ "approve": 0.2066,
+ "hold": 0.3032,
+ "manual_review": 0.335,
+ "reject": 0.1552
+ },
+ "confidence": 0.0311,
+ "answer_confidence": 0.335,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "matches_order": {
+ "type": "noul",
+ "noul": 0.5773,
+ "confidence": 0.5773,
+ "answer_confidence": 0.5773,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "duplicate": {
+ "type": "noul",
+ "noul": 0.15,
+ "confidence": 0.85,
+ "answer_confidence": 0.85,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "score": 1.5307,
+ "legend": {
+ "0": "None: everything reconciles.",
+ "1": "Trivial: rounding or a cosmetic difference.",
+ "2": "Moderate: a real difference worth confirming.",
+ "3": "Material: a large or unexplained difference."
+ },
+ "probabilities": {
+ "0": 0.3219,
+ "1": 0.1028,
+ "2": 0.2982,
+ "3": 0.2772
+ },
+ "confidence": 0.0513,
+ "answer_confidence": 0.3219,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgency": {
+ "type": "score",
+ "score": 1.5328,
+ "legend": {
+ "0": "No time pressure; can wait indefinitely.",
+ "1": "Routine; handle within the normal queue.",
+ "2": "Elevated; should be handled within the same week.",
+ "3": "Critical; requires action within the same day."
+ },
+ "probabilities": {
+ "0": 0.0409,
+ "1": 0.4738,
+ "2": 0.397,
+ "3": 0.0883
+ },
+ "confidence": 0.2313,
+ "answer_confidence": 0.4738,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 296,
+ "output_tokens": 0,
+ "state_tokens": 0,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ]
+ }
+ ]
+}
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/config/AgentConfig.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/config/AgentConfig.java
index cac69eb9..96e90291 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/config/AgentConfig.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/config/AgentConfig.java
@@ -17,10 +17,11 @@ import java.util.Map;
/**
* A checkpoint's {@code rl_agent_config.json}: sequence budgets and the fitted temperatures.
*
- * <p>These must be read from the checkpoint and never defaulted. The two shipped checkpoints
+ * <p>These must be read from the checkpoint and never defaulted. The three shipped checkpoints
* disagree on every one of them: english is {@code max_len 512 / head_max_len 192} with fitted
- * per-bucket temperatures, multilingual is {@code 1024/256} with none at all. A port that
- * hard-coded the library's own defaults would halve a multilingual state's budget.
+ * per-bucket temperatures, multilingual is {@code 1024/256} with none at all, and typed-decisions
+ * is {@code 1024/256} with a fitted table of its own. A port that hard-coded the library's own
+ * defaults would halve a multilingual state's budget.
*/
public final class AgentConfig {
@@ -28,9 +29,12 @@ public final class AgentConfig {
* A fitted temperature below 1 sharpens logits instead of softening them, and is refused.
*
* <p>Not a hypothetical: the english checkpoint ships {@code choice:11+} at <b>0.1006</b>, a
- * ~10x sharpener that publishes a 0.24 top probability as 0.99. A caller gating on confidence
- * would be told a coin flip is a certainty, so the runtime confines every temperature to
- * {@code [0.5, 5.0]}.
+ * ~10x sharpener that publishes a 0.24 top probability as 0.99, and typed-decisions carries
+ * the same value through from the checkpoint it was fine-tuned from. A caller gating on
+ * confidence would be told a coin flip is a certainty, so the runtime confines every
+ * temperature to {@code [0.5, 5.0]}. {@code TypedDecisionsParityTest} asserts the clamp
+ * end to end on a 14-option question, where skipping it sharpens the recorded 0.3035 top
+ * probability by roughly 5x without changing the argmax.
*/
public static final double TEMP_MIN = 0.5;
public static final double TEMP_MAX = 5.0;
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/Fixtures.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/Fixtures.java
index 2bb05820..c9334bdd 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/Fixtures.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/Fixtures.java
@@ -22,12 +22,28 @@ import java.util.Map;
*/
public final class Fixtures {
- /** Set to the directory holding {@code english/} and {@code multilingual/} to enable those tests. */
+ /**
+ * Set to the directory holding {@code english/}, {@code multilingual/} and
+ * {@code typed-decisions/} to enable those tests.
+ */
public static final String CHECKPOINTS_ENV = "LAYA_CHECKPOINTS";
/** Set to the directory holding {@code laya.onnx} (or {@code encoder.onnx} + {@code head.onnx}). */
public static final String GRAPH_ENV = "LAYA_ONNX_GRAPH";
+ /**
+ * As {@link #GRAPH_ENV}, for the graph traced from the {@code typed-decisions} checkpoint.
+ *
+ * <p>A second variable rather than reusing the first, because a graph carries the weights and
+ * nothing in it says which checkpoint traced it -- so a wrong pairing answers rather than
+ * failing. Measured: the english checkpoint opened against the typed-decisions graph raises
+ * nothing and returns {@code {refund: 0.9077, replace: 0.0923}}, digit for digit what the
+ * typed-decisions checkpoint returns against its own graph, and {@link Prediction#model()}
+ * reads {@code laya-rl-agent-onnx} either way. One variable holding one directory would make
+ * that pairing the default whenever both suites run in the same JVM.
+ */
+ public static final String TYPED_GRAPH_ENV = "LAYA_TYPED_ONNX_GRAPH";
+
/** Set to a {@code predict.json} recorded from Python with a graph present. */
public static final String PREDICT_GOLDEN_ENV = "LAYA_PREDICT_GOLDEN";
@@ -83,7 +99,7 @@ public final class Fixtures {
/** The message a skipped checkpoint test prints, so the reason is actionable. */
public static String missingCheckpoint(String name) {
return "needs the " + name + " checkpoint: set " + CHECKPOINTS_ENV
- + " to the directory holding english/ and multilingual/";
+ + " to the directory holding " + name + "/";
}
/** Helper: a list of longs from a fixture array. */
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/PredictParityTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/PredictParityTest.java
index 7abacf8a..e0a17c11 100644
--- a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/PredictParityTest.java
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/PredictParityTest.java
@@ -1,7 +1,9 @@
package com.convaiinnovations.laya;
+import static com.convaiinnovations.laya.PredictionGolden.assertAnswers;
+import static com.convaiinnovations.laya.PredictionGolden.assertUsage;
+import static com.convaiinnovations.laya.PredictionGolden.questionsOf;
import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertIterableEquals;
import static org.junit.jupiter.api.Assertions.assertTrue;
import java.io.IOException;
@@ -9,7 +11,6 @@ import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
-import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import org.junit.jupiter.api.AfterAll;
@@ -24,18 +25,11 @@ import org.junit.jupiter.api.TestFactory;
* <p>The only check that covers the whole stack at once -- tokenizer, sequence budgets, collation,
* graph I/O, decode, rounding and usage -- which the fixture families cannot do individually.
*
- * <p><b>Tolerance.</b> Structure is compared exactly: the answer type, the chosen label, the
- * question ids and their order, the legend, and every integer in {@link Usage}. Probabilities are
- * compared within {@value #PROBABILITY_TOLERANCE}, two units in the last reported decimal, because
- * the encoder's own arithmetic is platform-sensitive and the golden is recorded on one machine. A
- * chosen label is only asserted when the top two probabilities are clear of that tolerance; when
- * they are not, the argmax is genuinely a coin toss and asserting it would be asserting noise.
+ * <p>This one covers the multilingual checkpoint; {@link TypedDecisionsParityTest} covers
+ * {@code typed-decisions}. The comparison rules they share live in {@link PredictionGolden}.
*/
final class PredictParityTest {
- /** Two units in the last decimal an answer reports. */
- private static final double PROBABILITY_TOLERANCE = 2e-4;
-
/**
* One agent for the whole class, closed when the class is done.
*
@@ -108,121 +102,6 @@ final class PredictParityTest {
return tests;
}
- @SuppressWarnings("unchecked")
- private static void assertAnswers(Map<String, Object> want, Map<String, Answer> got) {
- assertIterableEquals(want.keySet(), got.keySet(), "question ids and their order");
- for (Map.Entry<String, Object> entry : want.entrySet()) {
- Map<String, Object> w = (Map<String, Object>) entry.getValue();
- Answer g = got.get(entry.getKey());
- String at = entry.getKey();
- assertEquals(w.get("type"), g.type(), at + ".type");
- assertEquals(number(w.get("answer_confidence")), g.answerConfidence(),
- PROBABILITY_TOLERANCE, at + ".answer_confidence");
- assertEquals(number(w.get("confidence")), g.confidence(),
- PROBABILITY_TOLERANCE, at + ".confidence");
- assertEquals(number(((Map<String, Object>) w.get("action")).get("act_probability")),
- g.actProbability(), PROBABILITY_TOLERANCE, at + ".act_probability");
- if (g instanceof Answer.Choice choice) {
- Map<String, Object> probabilities = (Map<String, Object>) w.get("probabilities");
- assertProbabilities(at, probabilities, choice.probabilities());
- if (decisive(probabilities)) {
- assertEquals(w.get("choice"), choice.choice(), at + ".choice");
- }
- } else if (g instanceof Answer.Score score) {
- assertEquals(number(w.get("score")), score.score(), PROBABILITY_TOLERANCE * 4,
- at + ".score");
- assertProbabilities(at, (Map<String, Object>) w.get("probabilities"),
- score.probabilities());
- Map<String, Object> legend = (Map<String, Object>) w.get("legend");
- assertIterableEquals(legend.keySet(), score.legend().keySet(), at + ".legend order");
- for (Map.Entry<String, Object> level : legend.entrySet()) {
- assertEquals(level.getValue(), score.legend().get(level.getKey()));
- }
- } else if (g instanceof Answer.Noul noul) {
- assertEquals(number(w.get("noul")), noul.noul(), PROBABILITY_TOLERANCE, at + ".noul");
- }
- }
- }
-
- /** Whether the top two probabilities are far enough apart for the argmax to be meaningful. */
- private static boolean decisive(Map<String, Object> probabilities) {
- double best = -1.0;
- double second = -1.0;
- for (Object value : probabilities.values()) {
- double p = number(value);
- if (p > best) {
- second = best;
- best = p;
- } else if (p > second) {
- second = p;
- }
- }
- return best - second > PROBABILITY_TOLERANCE * 5;
- }
-
- private static void assertProbabilities(String at, Map<String, Object> want,
- Map<String, Double> got) {
- assertIterableEquals(want.keySet(), got.keySet(), at + ".probability key order");
- for (Map.Entry<String, Object> entry : want.entrySet()) {
- assertEquals(number(entry.getValue()), got.get(entry.getKey()),
- PROBABILITY_TOLERANCE, at + ".p[" + entry.getKey() + "]");
- }
- }
-
- @SuppressWarnings("unchecked")
- private static void assertUsage(Map<String, Object> want, Usage got) {
- // Exact: these are counts, and a count that drifts is a bug, not float noise.
- assertEquals(((Number) want.get("input_tokens")).intValue(), got.inputTokens(),
- "usage.input_tokens");
- assertEquals(((Number) want.get("output_tokens")).intValue(), got.outputTokens());
- assertEquals(((Number) want.get("state_tokens")).intValue(), got.stateTokens(),
- "usage.state_tokens");
- assertEquals(((Number) want.get("state_tokens_dropped")).intValue(),
- got.stateTokensDropped(), "usage.state_tokens_dropped");
- assertEquals(want.get("truncated"), got.truncated(), "usage.truncated");
- assertIterableEquals((List<Object>) want.get("truncated_questions"),
- got.truncatedQuestions(), "usage.truncated_questions");
- Map<String, Object> options = (Map<String, Object>) want.get("options");
- if (options == null) {
- assertTrue(got.collapsedOptions().isEmpty(),
- "no collapse expected, got " + got.collapsedOptions().keySet());
- return;
- }
- assertEquals(options.keySet(), got.collapsedOptions().keySet(), "usage.options");
- for (Map.Entry<String, Object> entry : options.entrySet()) {
- Map<String, Object> w = (Map<String, Object>) entry.getValue();
- Usage.CollapsedOptions g = got.collapsedOptions().get(entry.getKey());
- assertEquals(((Number) w.get("total")).intValue(), g.total());
- assertEquals(((Number) w.get("distinct")).intValue(), g.distinct());
- }
- }
-
- private static double number(Object value) {
- return ((Number) value).doubleValue();
- }
-
- @SuppressWarnings("unchecked")
- private static Map<String, Question> questionsOf(Map<String, Object> defs) {
- Map<String, Question> out = new LinkedHashMap<>();
- for (Map.Entry<String, Object> entry : defs.entrySet()) {
- Map<String, Object> q = (Map<String, Object>) entry.getValue();
- String type = (String) q.get("type");
- String instructions = String.valueOf(q.get("instructions"));
- Object criteria = q.get("criteria");
- if ("choice".equals(type)) {
- out.put(entry.getKey(), Question.choice(instructions, (Map<String, Object>) criteria));
- } else if ("score".equals(type)) {
- out.put(entry.getKey(), Question.score(instructions, (List<Object>) criteria));
- } else {
- Map<String, Object> sides = criteria instanceof Map
- ? (Map<String, Object>) criteria : Map.of();
- out.put(entry.getKey(), Question.noul(instructions, sides.get("false"),
- sides.get("true"), null));
- }
- }
- return out;
- }
-
private static Map<String, Object> requireGolden() {
Map<String, Object> golden = Fixtures.load("predict.json");
Assumptions.assumeFalse(golden.containsKey("skipped"),
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/PredictionGolden.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/PredictionGolden.java
new file mode 100644
index 00000000..7c07d248
--- /dev/null
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/PredictionGolden.java
@@ -0,0 +1,148 @@
+package com.convaiinnovations.laya;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertIterableEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Comparing a recorded {@code predict} result against what this port answers.
+ *
+ * <p>Shared by {@link PredictParityTest} and {@link TypedDecisionsParityTest} rather than copied
+ * into each. The copy is the thing worth avoiding: the tolerance rules below are the gate those
+ * suites are, so two copies drifting apart means one checkpoint is held to a standard the other
+ * is not, and nothing reports it.
+ *
+ * <p><b>Tolerance.</b> Structure is compared exactly: the answer type, the chosen label, the
+ * question ids and their order, the legend, and every integer in {@link Usage}. Probabilities are
+ * compared within {@value #PROBABILITY_TOLERANCE}, two units in the last reported decimal, because
+ * the encoder's own arithmetic is platform-sensitive and the golden is recorded on one machine. A
+ * chosen label is only asserted when the top two probabilities are clear of that tolerance; when
+ * they are not, the argmax is genuinely a coin toss and asserting it would be asserting noise.
+ */
+final class PredictionGolden {
+
+ /** Two units in the last decimal an answer reports. */
+ static final double PROBABILITY_TOLERANCE = 2e-4;
+
+ private PredictionGolden() {
+ }
+
+ @SuppressWarnings("unchecked")
+ static void assertAnswers(Map<String, Object> want, Map<String, Answer> got) {
+ assertIterableEquals(want.keySet(), got.keySet(), "question ids and their order");
+ for (Map.Entry<String, Object> entry : want.entrySet()) {
+ Map<String, Object> w = (Map<String, Object>) entry.getValue();
+ Answer g = got.get(entry.getKey());
+ String at = entry.getKey();
+ assertEquals(w.get("type"), g.type(), at + ".type");
+ assertEquals(number(w.get("answer_confidence")), g.answerConfidence(),
+ PROBABILITY_TOLERANCE, at + ".answer_confidence");
+ assertEquals(number(w.get("confidence")), g.confidence(),
+ PROBABILITY_TOLERANCE, at + ".confidence");
+ assertEquals(number(((Map<String, Object>) w.get("action")).get("act_probability")),
+ g.actProbability(), PROBABILITY_TOLERANCE, at + ".act_probability");
+ if (g instanceof Answer.Choice choice) {
+ Map<String, Object> probabilities = (Map<String, Object>) w.get("probabilities");
+ assertProbabilities(at, probabilities, choice.probabilities());
+ if (decisive(probabilities)) {
+ assertEquals(w.get("choice"), choice.choice(), at + ".choice");
+ }
+ } else if (g instanceof Answer.Score score) {
+ assertEquals(number(w.get("score")), score.score(), PROBABILITY_TOLERANCE * 4,
+ at + ".score");
+ assertProbabilities(at, (Map<String, Object>) w.get("probabilities"),
+ score.probabilities());
+ Map<String, Object> legend = (Map<String, Object>) w.get("legend");
+ assertIterableEquals(legend.keySet(), score.legend().keySet(), at + ".legend order");
+ for (Map.Entry<String, Object> level : legend.entrySet()) {
+ assertEquals(level.getValue(), score.legend().get(level.getKey()));
+ }
+ } else if (g instanceof Answer.Noul noul) {
+ assertEquals(number(w.get("noul")), noul.noul(), PROBABILITY_TOLERANCE, at + ".noul");
+ }
+ }
+ }
+
+ /** Whether the top two probabilities are far enough apart for the argmax to be meaningful. */
+ static boolean decisive(Map<String, Object> probabilities) {
+ double best = -1.0;
+ double second = -1.0;
+ for (Object value : probabilities.values()) {
+ double p = number(value);
+ if (p > best) {
+ second = best;
+ best = p;
+ } else if (p > second) {
+ second = p;
+ }
+ }
+ return best - second > PROBABILITY_TOLERANCE * 5;
+ }
+
+ static void assertProbabilities(String at, Map<String, Object> want, Map<String, Double> got) {
+ assertIterableEquals(want.keySet(), got.keySet(), at + ".probability key order");
+ for (Map.Entry<String, Object> entry : want.entrySet()) {
+ assertEquals(number(entry.getValue()), got.get(entry.getKey()),
+ PROBABILITY_TOLERANCE, at + ".p[" + entry.getKey() + "]");
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ static void assertUsage(Map<String, Object> want, Usage got) {
+ // Exact: these are counts, and a count that drifts is a bug, not float noise.
+ assertEquals(((Number) want.get("input_tokens")).intValue(), got.inputTokens(),
+ "usage.input_tokens");
+ assertEquals(((Number) want.get("output_tokens")).intValue(), got.outputTokens());
+ assertEquals(((Number) want.get("state_tokens")).intValue(), got.stateTokens(),
+ "usage.state_tokens");
+ assertEquals(((Number) want.get("state_tokens_dropped")).intValue(),
+ got.stateTokensDropped(), "usage.state_tokens_dropped");
+ assertEquals(want.get("truncated"), got.truncated(), "usage.truncated");
+ assertIterableEquals((List<Object>) want.get("truncated_questions"),
+ got.truncatedQuestions(), "usage.truncated_questions");
+ Map<String, Object> options = (Map<String, Object>) want.get("options");
+ if (options == null) {
+ assertTrue(got.collapsedOptions().isEmpty(),
+ "no collapse expected, got " + got.collapsedOptions().keySet());
+ return;
+ }
+ assertEquals(options.keySet(), got.collapsedOptions().keySet(), "usage.options");
+ for (Map.Entry<String, Object> entry : options.entrySet()) {
+ Map<String, Object> w = (Map<String, Object>) entry.getValue();
+ Usage.CollapsedOptions g = got.collapsedOptions().get(entry.getKey());
+ assertEquals(((Number) w.get("total")).intValue(), g.total());
+ assertEquals(((Number) w.get("distinct")).intValue(), g.distinct());
+ }
+ }
+
+ static double number(Object value) {
+ return ((Number) value).doubleValue();
+ }
+
+ /** A recorded question mapping, rebuilt as the {@link Question} objects the port takes. */
+ @SuppressWarnings("unchecked")
+ static Map<String, Question> questionsOf(Map<String, Object> defs) {
+ Map<String, Question> out = new LinkedHashMap<>();
+ for (Map.Entry<String, Object> entry : defs.entrySet()) {
+ Map<String, Object> q = (Map<String, Object>) entry.getValue();
+ String type = (String) q.get("type");
+ String instructions = String.valueOf(q.get("instructions"));
+ Object criteria = q.get("criteria");
+ if ("choice".equals(type)) {
+ out.put(entry.getKey(), Question.choice(instructions, (Map<String, Object>) criteria));
+ } else if ("score".equals(type)) {
+ out.put(entry.getKey(), Question.score(instructions, (List<Object>) criteria));
+ } else {
+ Map<String, Object> sides = criteria instanceof Map
+ ? (Map<String, Object>) criteria : Map.of();
+ out.put(entry.getKey(), Question.noul(instructions, sides.get("false"),
+ sides.get("true"), null));
+ }
+ }
+ return out;
+ }
+}
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TypedDecisionsParityTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TypedDecisionsParityTest.java
new file mode 100644
index 00000000..2b8c9e92
--- /dev/null
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TypedDecisionsParityTest.java
@@ -0,0 +1,241 @@
+package com.convaiinnovations.laya;
+
+import static com.convaiinnovations.laya.PredictionGolden.assertAnswers;
+import static com.convaiinnovations.laya.PredictionGolden.assertUsage;
+import static com.convaiinnovations.laya.PredictionGolden.questionsOf;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import com.convaiinnovations.laya.config.AgentConfig;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.TreeSet;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.DynamicTest;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestFactory;
+
+/**
+ * End to end on the {@code typed-decisions} checkpoint, which nothing in this port used to run.
+ *
+ * <p>{@code Router} has always resolved the name and its four aliases, and {@link RouterTest}
+ * asserts that it does. What no test did was take the checkpoint those decisions name and push a
+ * state through it, so every number this port would produce from those weights was unmeasured --
+ * the routing decision was gated and its destination was not. This class closes that: the same
+ * comparison {@link PredictParityTest} applies to the multilingual checkpoint, applied to the
+ * weights {@code typed}, {@code typed_decisions}, {@code decisions} and
+ * {@code laya-typed-decisions} all resolve to.
+ *
+ * <p>What this checkpoint has that the multilingual one does not, and why each case is here:
+ *
+ * <ul>
+ * <li>Four fine-tuned workflow signatures. Each recorded case names the workflow its question
+ * ids are, and the workflow is re-derived here through
+ * {@link Router#matchTypedDecisionsWorkflow} -- so the routing decision and the forward
+ * pass are asserted on one object instead of in two suites that never meet.
+ * <li>A populated {@code temperature_by_options}, where the multilingual checkpoint ships an
+ * empty one. The {@code many-options} case has 14 options, which lands in the
+ * {@code choice:11+} bucket this checkpoint ships at 0.1006 -- outside the accepted
+ * [0.5, 5] band, so it is clamped to 0.5. Skipping the clamp sharpens the recorded top
+ * probability of 0.3035 by roughly 5x and still returns the same argmax, so the
+ * probabilities are the only thing that separates the two implementations.
+ * <li>{@code max_len} 1024 against the english checkpoint's 512. The {@code over-budget} case
+ * drops 1211 state tokens at that budget and marks all five questions truncated.
+ * </ul>
+ */
+final class TypedDecisionsParityTest {
+
+ /** The checkpoint this golden was recorded from; also the directory it lives in. */
+ private static final String CHECKPOINT = "typed-decisions";
+
+ /** Shared for the class: the graph is 1.7 GB of weights and opening it twice buys nothing. */
+ private static Agent shared;
+
+ @AfterAll
+ static void closeAgent() {
+ if (shared != null) {
+ shared.close();
+ shared = null;
+ }
+ }
+
+ /**
+ * The loaded config is the typed-decisions one, before any probability is compared.
+ *
+ * <p>Not ceremony. This checkpoint is fine-tuned FROM the english one and ships its
+ * {@code tokenizer.json} byte for byte and its {@code answerdotai/ModernBERT-large} encoder,
+ * so the english checkpoint opened against this graph raises nothing and answers
+ * {@code {refund: 0.9077, replace: 0.0923}} -- the same digits as the right pairing, because
+ * the weights are the graph's. {@link Prediction#model()} does not separate them either: the
+ * ONNX path reports {@code laya-rl-agent-onnx} whichever checkpoint it opened. The budgets
+ * and the temperatures do separate them, so they are what is asserted.
+ */
+ @Test
+ @DisplayName("the agent under test is the typed-decisions checkpoint, not the english one")
+ @SuppressWarnings("unchecked")
+ void configIsTheTypedCheckpoint() throws IOException {
+ Map<String, Object> golden = requireGolden();
+ Map<String, Object> want = (Map<String, Object>) golden.get("config");
+ AgentConfig config = agent(golden).config();
+ assertEquals(want.get("encoder"), config.encoder(), "encoder");
+ assertEquals(((Number) want.get("max_len")).intValue(), config.maxLen(), "max_len");
+ assertEquals(((Number) want.get("head_max_len")).intValue(), config.headMaxLen(),
+ "head_max_len");
+ List<Object> temperature = (List<Object>) want.get("temperature");
+ assertEquals(temperature.size(), config.temperature().length, "temperature length");
+ for (int i = 0; i < temperature.size(); i++) {
+ // Exact: these are read out of a JSON file, not computed, so any difference is a
+ // parse bug rather than float noise.
+ assertEquals(((Number) temperature.get(i)).doubleValue(), config.temperature()[i],
+ "temperature[" + i + "]");
+ }
+ Map<String, Object> buckets = (Map<String, Object>) want.get("temperature_by_options");
+ assertEquals(buckets.keySet(), config.temperatureByOptions().keySet(),
+ "temperature_by_options buckets");
+ for (Map.Entry<String, Object> entry : buckets.entrySet()) {
+ assertEquals(((Number) entry.getValue()).doubleValue(),
+ config.temperatureByOptions().get(entry.getKey()),
+ "temperature_by_options[" + entry.getKey() + "]");
+ }
+ // The clamp, asserted on the one bucket that needs it rather than inferred from the map
+ // above. 0.1006 in `rl_agent_config.json`, 0.5 in use: a port that read the file value
+ // straight through would sharpen the many-options distribution by ~5x, and a lone 0.5 in
+ // the golden could not be told from a bucket that was fitted at 0.5.
+ Map<String, Object> raw = (Map<String, Object>) want.get("temperature_by_options_raw");
+ assertEquals(0.10058280825614929, ((Number) raw.get("choice:11+")).doubleValue(),
+ "the checkpoint still ships the sharpening bucket this case depends on");
+ assertEquals(AgentConfig.TEMP_MIN, config.temperatureByOptions().get("choice:11+"),
+ "choice:11+ must be clamped to TEMP_MIN, not applied as fitted");
+ }
+
+ /**
+ * Every recorded workflow id set still routes to the workflow it was recorded under.
+ *
+ * <p>The link MODELS.md said was missing. A signature that stops matching would otherwise
+ * leave the forward-pass cases below passing under a workflow name the router no longer
+ * returns, which reads as coverage of a path that no longer exists.
+ */
+ @Test
+ @DisplayName("each case's question ids still match the workflow it was recorded under")
+ @SuppressWarnings("unchecked")
+ void workflowsStillMatch() {
+ Map<String, Object> golden = requireGolden();
+ Map<String, Object> workflows = (Map<String, Object>) golden.get("workflows");
+ assertEquals(4, workflows.size(), "the checkpoint is fine-tuned on four workflows");
+ // DISTINCT, not a count of cases: two cases share the invoice_processing signature, so
+ // counting cases made four workflows look like five and the floor failed on arithmetic
+ // instead of on coverage.
+ Set<String> covered = new TreeSet<>();
+ for (Map.Entry<String, Object> entry
+ : ((Map<String, Object>) golden.get("single")).entrySet()) {
+ Map<String, Object> c = (Map<String, Object>) entry.getValue();
+ Map<String, Object> questions = (Map<String, Object>) c.get("questions");
+ String want = (String) c.get("workflow");
+ assertEquals(want, Router.matchTypedDecisionsWorkflow(questions),
+ entry.getKey() + ": workflow for ids " + questions.keySet());
+ if (want != null) {
+ assertTrue(workflows.containsKey(want), want + " is not one of the four");
+ covered.add(want);
+ }
+ }
+ assertEquals(workflows.keySet(), covered,
+ "every workflow needs a recorded forward pass");
+ }
+
+ @TestFactory
+ @DisplayName("single-state typed-decisions predictions match Python")
+ @SuppressWarnings("unchecked")
+ List<DynamicTest> singleMatches() throws IOException {
+ Map<String, Object> golden = requireGolden();
+ Map<String, Object> singles = (Map<String, Object>) golden.get("single");
+ Agent agent = agent(golden);
+ List<DynamicTest> tests = new ArrayList<>();
+ for (Map.Entry<String, Object> entry : singles.entrySet()) {
+ Map<String, Object> c = (Map<String, Object>) entry.getValue();
+ tests.add(DynamicTest.dynamicTest(entry.getKey(), () -> {
+ Prediction got = agent.predict(c.get("state"),
+ questionsOf((Map<String, Object>) c.get("questions")),
+ (String) c.get("lang"));
+ assertEquals(c.get("model"), got.model());
+ assertAnswers((Map<String, Object>) c.get("answers"), got.answers());
+ assertUsage((Map<String, Object>) c.get("usage"), got.usage());
+ }));
+ }
+ // Four workflows plus the three cases that are here for the checkpoint's own budgets and
+ // temperature buckets. A factory whose fixture key emptied used to generate zero
+ // assertions and still exit 0, which is what this floor is for.
+ assertTrue(tests.size() >= 8, "expected the golden's cases, got " + tests.size());
+ return tests;
+ }
+
+ @TestFactory
+ @DisplayName("batched typed-decisions predictions match Python at every batch size")
+ @SuppressWarnings("unchecked")
+ List<DynamicTest> batchMatches() throws IOException {
+ Map<String, Object> golden = requireGolden();
+ Agent agent = agent(golden);
+ List<DynamicTest> tests = new ArrayList<>();
+ for (Object configObject : (List<Object>) golden.get("batch")) {
+ Map<String, Object> config = (Map<String, Object>) configObject;
+ Object size = config.get("batch_size");
+ boolean sort = Boolean.TRUE.equals(config.get("sort_by_length"));
+ tests.add(DynamicTest.dynamicTest(
+ "batch_size=" + (size == null ? "all" : size) + " sort=" + sort, () -> {
+ List<Prediction> got = agent.predictBatch(
+ (List<Object>) config.get("states"),
+ questionsOf((Map<String, Object>) config.get("questions")),
+ null, size == null ? 0 : ((Number) size).intValue(), sort);
+ List<Object> want = (List<Object>) config.get("results");
+ assertEquals(want.size(), got.size(), "one prediction per state");
+ for (int i = 0; i < want.size(); i++) {
+ Map<String, Object> w = (Map<String, Object>) want.get(i);
+ // The caller's order, whatever order the batcher grouped them in.
+ assertAnswers((Map<String, Object>) w.get("answers"),
+ got.get(i).answers());
+ assertUsage((Map<String, Object>) w.get("usage"), got.get(i).usage());
+ }
+ }));
+ }
+ assertTrue(tests.size() >= 3, "expected the golden's batch configs, got " + tests.size());
+ return tests;
+ }
+
+ private static Map<String, Object> requireGolden() {
+ Map<String, Object> golden = Fixtures.load("typed_decisions.json");
+ Assumptions.assumeFalse(golden.containsKey("skipped"),
+ "typed_decisions.json was recorded without a graph: set " + Fixtures.TYPED_GRAPH_ENV
+ + " and re-run scripts/gen_fixtures.py");
+ return golden;
+ }
+
+ /** The shared agent, opened on first use so the assumptions still skip cleanly. */
+ private static synchronized Agent agent(Map<String, Object> golden) throws IOException {
+ if (shared == null) {
+ shared = openAgent(golden);
+ }
+ return shared;
+ }
+
+ private static Agent openAgent(Map<String, Object> golden) throws IOException {
+ String checkpoint = (String) golden.getOrDefault("checkpoint", CHECKPOINT);
+ Path model = Fixtures.checkpoint(checkpoint);
+ Assumptions.assumeTrue(model != null, Fixtures.missingCheckpoint(checkpoint));
+ String graph = System.getenv(Fixtures.TYPED_GRAPH_ENV);
+ Assumptions.assumeTrue(graph != null && !graph.isBlank(),
+ "set " + Fixtures.TYPED_GRAPH_ENV + " to the laya.onnx exported from the "
+ + checkpoint + " checkpoint");
+ Path graphPath = Paths.get(graph);
+ Path graphDirectory = Files.isDirectory(graphPath) ? graphPath : graphPath.getParent();
+ Assumptions.assumeTrue(graphDirectory != null && Files.isDirectory(graphDirectory),
+ Fixtures.TYPED_GRAPH_ENV + " does not name a graph: " + graph);
+ return Agent.open(model, graphDirectory);
+ }
+}
diff --git a/laya-java/scripts/gen_fixtures.py b/laya-java/scripts/gen_fixtures.py
index 1b335cb9..26f66aa2 100644
--- a/laya-java/scripts/gen_fixtures.py
+++ b/laya-java/scripts/gen_fixtures.py
@@ -1817,6 +1817,429 @@ def predict_golden():
"single": single, "batch": batches}
+# -- the typed-decisions checkpoint ---------------------------------------------------------
+#
+# `laya-typed-decisions` is fine-tuned from the ENGLISH checkpoint, which is what makes a
+# fixture family for it worth its weight. It ships the same `tokenizer.json` byte for byte
+# (diffed: identical) and the same `answerdotai/ModernBERT-large` encoder, and nothing in a
+# `.onnx` file says which checkpoint traced it. Measured: opening the ENGLISH checkpoint against
+# the typed graph raises nothing and answers {refund: 0.9077, replace: 0.0923} on a two-option
+# question -- the same digits, to the last one, as the typed checkpoint against its own graph,
+# because the weights are entirely the graph's and the two configs agree on the `choice:2`
+# temperature. `model` does not separate them either: the ONNX path reports
+# `laya-rl-agent-onnx` whichever checkpoint it opened. Only the multilingual checkpoint fails
+# loudly, and for an unrelated reason -- a 235292 token id against a 50368-row embedding, which
+# ONNX Runtime rejects at the `node_embedding` Gather.
+#
+# So the discriminator has to come from `rl_agent_config.json`, which is where the two do differ:
+# `max_len` 1024 against english's 512, `head_max_len` 256 against 192, `model_name`
+# `laya-typed-decisions` against `rl-agent`, and base temperatures [1.0148, 1.0374, 1.0575]
+# against [1.6369, 1.2514, 1.9834]. Those are recorded below as `config`, and the Java test
+# asserts them before it compares a single probability.
+#
+# The question ids are read from `laya.router` rather than retyped: an exact id-set match is the
+# entire signature `match_typed_decisions_workflow` tests, so a signature change has to move this
+# family rather than leave it asserting a workflow name the router no longer returns.
+
+# `invoice_processing` is the one workflow whose TRAINED schema is in this repository, in
+# `examples/32_typed_decisions_workflow.py` (dataset `LocalLLaMA/typed-decisions`, config
+# `invoice_processing`). It is duplicated here rather than imported because that example has a
+# module-level `agent = load("typed-decisions")`, so importing it would download 846 MB and run
+# the model; `_typed_questions` below fails the generator if the ids ever stop matching the
+# router's signature, which is the drift that would matter.
+TYPED_INVOICE_QUESTIONS = {
+ "disposition": {
+ "type": "choice",
+ "instructions": "How should this vendor invoice be dispositioned?",
+ "criteria": {
+ "approve": "Matches the order and delivery; approve for payment.",
+ "hold": "Something needs confirming before payment; hold pending clarification.",
+ "manual_review": "A human in finance must review the discrepancy.",
+ "reject": "Should not be paid: duplicate, unauthorised or materially wrong.",
+ },
+ },
+ "matches_order": {
+ "type": "noul",
+ "instructions": "The invoice reconciles with the purchase order and the recorded delivery.",
+ "criteria": {
+ "false": "There is a discrepancy against the order or the delivery.",
+ "true": "Line items, quantities and amounts reconcile.",
+ },
+ },
+ "duplicate": {
+ "type": "noul",
+ "instructions": "This invoice appears to duplicate an invoice already submitted.",
+ },
+ "discrepancy_severity": {
+ "type": "score",
+ "instructions": "How material is any discrepancy between the invoice, the order "
+ "and the delivery?",
+ "criteria": [
+ "None: everything reconciles.",
+ "Trivial: rounding or a cosmetic difference.",
+ "Moderate: a real difference worth confirming.",
+ "Material: a large or unexplained difference.",
+ ],
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How time-sensitive is processing this invoice?",
+ "criteria": [
+ "No time pressure; can wait indefinitely.",
+ "Routine; handle within the normal queue.",
+ "Elevated; should be handled within the same week.",
+ "Critical; requires action within the same day.",
+ ],
+ },
+}
+
+# The other three workflows' TRAINED instruction text is not in this repository -- only their id
+# signatures are, in `laya.router._TYPED_DECISION_WORKFLOWS`. These schemas therefore carry the
+# real ids and instruction text written here, which is stated rather than glossed over: what the
+# fixture gates is the forward pass and the routing decision that reaches it, and both are real
+# for any schema whose ids match. The text is recorded verbatim in the fixture, so a port renders
+# the identical prompt and must reproduce the identical answer.
+TYPED_CUSTOMER_SERVICE_QUESTIONS = {
+ "action": {
+ "type": "choice",
+ "instructions": "What should the support team do next?",
+ "criteria": {
+ "refund": "Return the money and close the ticket.",
+ "replace": "Ship a replacement unit.",
+ "explain": "Answer the question; nothing to send.",
+ "escalate": "Hand to a senior agent.",
+ },
+ },
+ "category": {
+ "type": "choice",
+ "instructions": "Which queue does this ticket belong in?",
+ "criteria": {
+ "billing": "Charges, invoices and refunds.",
+ "shipping": "Delivery, tracking and damage in transit.",
+ "account": "Login, access and account settings.",
+ },
+ },
+ "churn_risk": {
+ "type": "score",
+ "instructions": "How likely is this customer to leave over this?",
+ "criteria": [
+ "Not at risk; routine contact.",
+ "Mildly annoyed.",
+ "Considering alternatives.",
+ "Explicitly threatening to cancel.",
+ ],
+ },
+ "needs_human": {
+ "type": "noul",
+ "instructions": "This ticket needs a human agent rather than an automated reply.",
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How quickly must this be answered?",
+ "criteria": [
+ "No time pressure.",
+ "Within the normal queue.",
+ "Same week.",
+ "Same day.",
+ ],
+ },
+}
+
+TYPED_SECURITY_QUESTIONS = {
+ "credential_compromise": {
+ "type": "noul",
+ "instructions": "Credentials belonging to this account were exposed or reused.",
+ "criteria": {
+ "false": "No sign that a secret left the organisation.",
+ "true": "A password, token or key is in the wrong hands.",
+ },
+ },
+ "disposition": {
+ "type": "choice",
+ "instructions": "How should this alert be dispositioned?",
+ "criteria": {
+ "close": "Benign; close with no action.",
+ "monitor": "Watch for a repeat before acting.",
+ "contain": "Isolate the host or revoke the credential now.",
+ "escalate": "Wake an incident responder.",
+ },
+ },
+ "severity": {
+ "type": "score",
+ "instructions": "How severe is the impact if this is real?",
+ "criteria": [
+ "Informational.",
+ "Low: one account, no data.",
+ "High: production data or lateral movement.",
+ "Critical: active, spreading, or customer data.",
+ ],
+ },
+ "true_positive": {
+ "type": "noul",
+ "instructions": "This alert describes genuine malicious activity rather than noise.",
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How quickly must someone respond?",
+ "criteria": [
+ "No time pressure.",
+ "Next business day.",
+ "Within the hour.",
+ "Immediately.",
+ ],
+ },
+}
+
+TYPED_AGENT_TRACE_QUESTIONS = {
+ "action": {
+ "type": "choice",
+ "instructions": "What should be done with this agent run?",
+ "criteria": {
+ "accept": "The run did what was asked; accept it.",
+ "retry": "Transient failure; run it again.",
+ "rollback": "Undo what it changed.",
+ "escalate": "A person must look at this.",
+ },
+ },
+ "needs_review": {
+ "type": "noul",
+ "instructions": "A human should read this trace before the result is used.",
+ },
+ "outcome": {
+ "type": "choice",
+ "instructions": "How did the run end?",
+ "criteria": {
+ "success": "Completed the task.",
+ "partial": "Completed some of it.",
+ "failure": "Did not complete the task.",
+ },
+ },
+ "risk": {
+ "type": "score",
+ "instructions": "How much damage could this run's actions have caused?",
+ "criteria": [
+ "Read-only; no effect.",
+ "Reversible writes.",
+ "Irreversible writes to internal systems.",
+ "Irreversible, customer-visible, or spending money.",
+ ],
+ },
+ "urgency": {
+ "type": "score",
+ "instructions": "How quickly must this be looked at?",
+ "criteria": [
+ "No time pressure.",
+ "Next business day.",
+ "Within the hour.",
+ "Immediately.",
+ ],
+ },
+}
+
+# The two states from `examples/32_typed_decisions_workflow.py`: one with a 40% unit-price
+# variance against the purchase order, one that reconciles exactly. They are a pair on purpose --
+# the same schema on two states that differ only in the numbers is what shows the forward pass is
+# reading the state rather than the questions.
+TYPED_INVOICE_VARIANCE = {
+ "invoice": {
+ "id": "INV-2026-8842", "vendor": "Northwind Castings", "currency": "USD",
+ "total_usd": 300020.0,
+ "lines": [{"sku": "SKU-940", "qty": 1000, "unit_usd": 300.02, "total_usd": 300020.0}],
+ },
+ "purchase_order": {
+ "id": "PO-3392", "total_usd": 214300.0,
+ "lines": [{"qty": 1000, "unit_usd": 214.3}],
+ "freight_terms": "freight prepaid by vendor, not separately billable",
+ },
+ "delivery": {"date": "2026-03-18", "received_qty": 1000,
+ "condition": "accepted with exceptions"},
+ "payment": {"terms": "net 45", "status": "overdue", "days_until_due": -3},
+ "vendor_history": {
+ "invoices_12m": 40, "disputes_12m": 0,
+ "prior_invoice_ids": ["INV-2026-6633", "INV-2026-9563", "INV-2026-7939"],
+ },
+}
+
+TYPED_INVOICE_CLEAN = {
+ "invoice": {
+ "id": "INV-2026-9107", "vendor": "Cedar Freight", "currency": "USD",
+ "total_usd": 18450.0,
+ "lines": [{"sku": "SKU-221", "qty": 300, "unit_usd": 61.5, "total_usd": 18450.0}],
+ },
+ "purchase_order": {
+ "id": "PO-3477", "total_usd": 18450.0,
+ "lines": [{"qty": 300, "unit_usd": 61.5}],
+ "freight_terms": "freight prepaid by vendor, not separately billable",
+ },
+ "delivery": {"date": "2026-03-20", "received_qty": 300, "condition": "accepted in full"},
+ "payment": {"terms": "net 30", "status": "current", "days_until_due": 18},
+ "vendor_history": {"invoices_12m": 12, "disputes_12m": 0,
+ "prior_invoice_ids": ["INV-2026-7781"]},
+}
+
+# 14 options, which is what puts the `choice:11+` temperature bucket in play. This checkpoint
+# ships that bucket at 0.1006 -- outside the [0.5, 5] band `laya.common` accepts, so it is clamped
+# to 0.5 and the answer is published uncalibrated. A port that reads the bucket and skips the
+# clamp sharpens this distribution by ~5x and still returns a plausible-looking argmax, so the
+# probabilities here are the only thing that tells the two apart.
+TYPED_MANY_OPTION_QUESTIONS = {
+ "band": {"type": "choice", "instructions": "Which spend band does this invoice fall in?",
+ "criteria": {("band%02d" % i): ("tier number %d" % i) for i in range(14)}},
+}
+
+# `max_len` is 1024 here against the english checkpoint's 512, so a state that truncates on one
+# does not truncate on the other. This one is long enough to drop tokens at 1024, which is what
+# makes `state_tokens_dropped` and `truncated` non-zero rather than decorative.
+TYPED_OVER_BUDGET_STATE = ("line item number one, unit price 300.02 against a purchase order at "
+ "214.30, delivery accepted with exceptions. ") * 90
+
+# A question set whose ids are NOT a workflow signature. The checkpoint still answers it -- being
+# fine-tuned on four schemas does not make it refuse a fifth -- and recording that is what stops
+# a port from special-casing the signatures somewhere in the forward pass, where they have no
+# business being. Routing is the only place the signature is read.
+TYPED_OFF_WORKFLOW_QUESTIONS = {
+ "department": {"type": "choice", "instructions": "Which team owns this?",
+ "criteria": {"billing": "charges and refunds", "other": "anything else"}},
+}
+
+
+def _typed_questions(workflow, questions):
+ """`questions`, after checking its ids still are `workflow`'s signature.
+
+ The schemas above are literals, and the signatures they have to match live in `laya.router`.
+ Nothing else connects the two, so without this the router could gain a question id and this
+ family would carry on recording a forward pass under a workflow name
+ `match_typed_decisions_workflow` no longer returns -- a fixture that names a routing outcome
+ it does not produce, which is worse than no fixture.
+ """
+ from laya.router import _TYPED_DECISION_WORKFLOWS, match_typed_decisions_workflow
+
+ expected = _TYPED_DECISION_WORKFLOWS[workflow]
+ got = match_typed_decisions_workflow(questions)
+ if got != workflow:
+ raise SystemExit(
+ "gen_fixtures: the %s schema's ids are %s, but laya.router's signature is %s "
+ "(match_typed_decisions_workflow returned %r)"
+ % (workflow, sorted(questions), sorted(expected), got))
+ return questions
+
+
+# (case id, state, lang, workflow or None, questions). `workflow` is recorded so the Java test can
+# assert the routing decision and the forward pass on the SAME object: MODELS.md's gap was that
+# `Router` resolving `typed-decisions` was tested while nothing ever ran the checkpoint, so a
+# fixture that records one without the other leaves half of it open.
+TYPED_CASES = [
+ ("invoice-variance", TYPED_INVOICE_VARIANCE, None, "invoice_processing",
+ TYPED_INVOICE_QUESTIONS),
+ ("invoice-clean", TYPED_INVOICE_CLEAN, None, "invoice_processing", TYPED_INVOICE_QUESTIONS),
+ ("customer-service", "We were billed twice for March. Third time this year -- refund it "
+ "today or we are moving to a competitor.", None, "customer_service",
+ TYPED_CUSTOMER_SERVICE_QUESTIONS),
+ ("security-incident", {"alert": "impossible travel", "user": "[email protected]",
+ "signins": [{"city": "Lagos", "at": "2026-03-18T09:02Z"},
+ {"city": "Vilnius", "at": "2026-03-18T09:41Z"}],
+ "mfa": "satisfied by push, approved on the second prompt",
+ "prior_alerts_30d": 0}, None, "security_incidents",
+ TYPED_SECURITY_QUESTIONS),
+ ("agent-trace", {"run": "agt-7741", "tool_calls": 14, "wrote": ["s3://prod-reports/q1.csv"],
+ "errors": ["RateLimit on call 9, retried"],
+ "final": "uploaded the report, but the row count is 0"},
+ None, "agent_trace_observability", TYPED_AGENT_TRACE_QUESTIONS),
+ ("many-options", "The invoice totals 300,020 USD against a 214,300 USD order.", None, None,
+ TYPED_MANY_OPTION_QUESTIONS),
+ # Same ids as the two invoice cases, so the same workflow -- recording this as "no workflow"
+ # because the state is different was the first thing the Java test caught.
+ ("over-budget", TYPED_OVER_BUDGET_STATE, None, "invoice_processing",
+ TYPED_INVOICE_QUESTIONS),
+ ("off-workflow", "We were billed twice for March and want a refund today.", "en", None,
+ TYPED_OFF_WORKFLOW_QUESTIONS),
+]
+
+
+def typed_decisions_golden():
+ """`ONNXAgent.predict` and `predict_batch` on the `typed-decisions` checkpoint.
+
+ The sibling of `predict_golden`, against the third checkpoint. It is a separate family and a
+ separate graph because a graph is traced from ONE checkpoint and `LAYA_ONNX_GRAPH` names one
+ directory: pointing that at the typed graph while `predict.json` expects the multilingual one
+ would re-record `predict.json` from the wrong weights, and both sides of that comparison come
+ from the same mistake, so it passes. `LAYA_TYPED_ONNX_GRAPH` keeps the two unable to collide.
+
+ Records `skipped` when the graph is absent, so `unverifiable` leaves the committed file alone
+ rather than replacing it with a marker.
+
+ Platform-sensitive in the last reported digit, as `predict_golden` is; the Java test's
+ tolerance is the gate, not byte equality of this file.
+ """
+ import os
+
+ graph = os.environ.get("LAYA_TYPED_ONNX_GRAPH")
+ rig = os.environ.get("LAYA_FIXTURE_CHECKPOINTS", DEFAULT_CHECKPOINT_ROOT)
+ model = "typed-decisions"
+ if not graph or not os.path.exists(graph):
+ return {"skipped": "set LAYA_TYPED_ONNX_GRAPH to a laya.onnx exported from the "
+ "typed-decisions checkpoint to record this family"}
+ from laya.onnx_agent import ONNXAgent
+ from laya.router import _TYPED_DECISION_WORKFLOWS
+
+ agent = ONNXAgent(os.path.join(rig, model), onnx_path=graph)
+ cfg = agent.cfg
+ single = {}
+ for cid, state, lang, workflow, questions in TYPED_CASES:
+ if workflow is not None:
+ questions = _typed_questions(workflow, questions)
+ result = agent.predict(state, questions, lang=lang)
+ single[cid] = {"state": state, "lang": lang, "workflow": workflow,
+ "questions": questions, "model": result["model"],
+ "answers": result["answers"], "usage": result["usage"]}
+
+ # Batched over the real invoice schema, including the empty state and two short ones, so
+ # collation and padding are exercised here too. The empty state is the one that matters: it
+ # contributes 0 state tokens and still costs 296 input tokens of rendered questions, so a
+ # port that skips an empty member rather than padding it gets a different batch shape.
+ batch_states = [TYPED_INVOICE_VARIANCE, TYPED_INVOICE_CLEAN,
+ "INV-2026-1004, 980 USD, matches PO-3100 and delivery.",
+ "duplicate of INV-2026-1004", ""]
+ batch_questions = _typed_questions("invoice_processing", TYPED_INVOICE_QUESTIONS)
+ batches = []
+ for batch_size, sort_by_length in ((None, False), (2, False), (3, True)):
+ kwargs = {"sort_by_length": sort_by_length}
+ if batch_size is not None:
+ kwargs["batch_size"] = batch_size
+ results = agent.predict_batch(batch_states, batch_questions, **kwargs)
+ batches.append({"batch_size": batch_size, "sort_by_length": sort_by_length,
+ "states": batch_states, "questions": batch_questions,
+ "results": [{"model": r["model"], "answers": r["answers"],
+ "usage": r["usage"]} for r in results]})
+
+ return {
+ "checkpoint": model,
+ "graph": os.path.basename(graph),
+ # What tells this checkpoint from the english one it was fine-tuned from, which ships the
+ # identical tokenizer and the identical encoder. Asserted by the Java test before any
+ # probability is compared, because a mismatch here means the numbers below were produced
+ # by the wrong weights and comparing them is worse than not running.
+ "config": {
+ "model_name": cfg["model_name"],
+ "encoder": cfg["encoder"],
+ "max_len": cfg["max_len"],
+ "head_max_len": cfg["head_max_len"],
+ # `agent.temperature*`, not `cfg[...]`: the agent clamps on load and the config
+ # dict keeps the file's value, so the two disagree here and a port exposes only the
+ # clamped one. Recording both is what makes the clamp visible -- a lone 0.5 cannot
+ # be told from a temperature that was fitted at 0.5, and this checkpoint's
+ # `choice:11+` is 0.1006 in the file and 0.5 in use.
+ "temperature": list(agent.temperature),
+ "temperature_by_options": dict(agent.temperature_by_options),
+ "temperature_raw": list(agent.temperature_raw),
+ "temperature_by_options_raw": dict(agent.temperature_by_options_raw),
+ },
+ "workflows": {name: sorted(ids) for name, ids in sorted(_TYPED_DECISION_WORKFLOWS.items())},
+ "single": single,
+ "batch": batches,
+ }
+
+
def email_clean():
"""`laya.email`'s cleaner and state builder, over a corpus built from its own reasoning.
@@ -3105,6 +3528,7 @@ FAMILIES = {
"python_json.json": python_json,
"email.json": email_clean,
"predict.json": predict_golden,
+ "typed_decisions.json": typed_decisions_golden,
}