Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(java): port predict_tournament as Shortlist.predictTournament
Adeshina Lasisi71e43b75e5f7
diff --git a/.github/workflows/java.yml b/.github/workflows/java.ymlindex 5b65c99f..615161a3 100644--- a/.github/workflows/java.yml+++ b/.github/workflows/java.yml@@ -154,17 +154,17 @@ jobs:# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest# run. `--min-tests` is what stops an aborted suite from looking like a pass.#- # Measured on the merged tree: 980 register here, 963 run, 17 abort on a checkpoint this+ # Measured on the merged tree: 1133 register here, 1113 run, 20 abort on a checkpoint this# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes- # `TestJvmVersionTest` run instead of skip, so it executes 964. Same suite, one more test,+ # `TestJvmVersionTest` run instead of skip, so it executes 1114. Same suite, one more test,# because the flag enables it. The step below is where the number is justified.- name: Tests that need no modelworking-directory: laya-javarun: ./gradlew --no-daemon test- # Measured on the merged tree, Corretto 17, no checkpoints: 980 registered, 17 abort,- # 963 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in+ # Measured on the merged tree, Corretto 17, no checkpoints: 1133 registered, 20 abort,+ # 1113 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this# replaced. A margin only binds classes larger than itself, and three classes in this lane# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no@@ -179,7 +179,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 963 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 1113 --allow-aborted# The samples are single-file source programs, deliberately NOT a Gradle module: a# `samples` subproject would inherit the root build's MavenPublication and@@ -335,7 +335,7 @@ jobs:# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice# and reported it as two JDKs. With `--require-passed` it exits 1.#- # 964, not the build lane's 963, and the one is the point: this lane runs one test MORE+ # 1114, not the build lane's 1113, and the one is the point: this lane runs one test MORE# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --# three classes here run exactly one test, so no positive margin can bind them.@@ -356,7 +356,7 @@ jobs:if: ${{ !cancelled() }}run: >python laya-java/scripts/check_test_results.py- laya-java/laya-java/build/test-results/test --min-tests 964 --allow-aborted+ laya-java/laya-java/build/test-results/test --min-tests 1114 --allow-aborted--require-passed TestJvmVersionTest- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2if: ${{ failure() }}@@ -459,9 +459,12 @@ jobs:- name: predictcheckpoints: multilingualgraph-model: multilingual- families: --only predict.json- tests: --tests com.convaiinnovations.laya.PredictParityTest- min-tests: 9+ families: --only predict.json --only tournament_golden.json+ tests: >-+ --tests com.convaiinnovations.laya.PredictParityTest+ --tests com.convaiinnovations.laya.TournamentParityTest+ # Measured locally: 9 PredictParityTest + 3 TournamentParityTest. Exact.+ min-tests: 12timeout: 75# Weight-bearing, and a SEPARATE cell rather than a second family in the one above.# Three reasons, in order of how much they cost:@@ -573,7 +576,7 @@ jobs:# Re-recorded from the Python code at THIS commit. `--strict` makes a family this cell could# not regenerate an error rather than a skip: here a skip means the download or the export# failed, not that the machine has no model. It is scoped by `--only` to the families this- # cell owns, and between the two cells all four checkpoint-backed families are covered.+ # cell owns, and between the cells every checkpoint-backed family is covered.- name: Regenerate this cell's fixtures from Pythonenv:LAYA_FIXTURE_CHECKPOINTS: ${{ github.workspace }}/laya-java/.work/checkpointsdiff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.ymlindex c9e6629f..774a74b1 100644--- a/.github/workflows/release-java.yml+++ b/.github/workflows/release-java.yml@@ -67,7 +67,7 @@ jobs:# that lane already covered.- name: Model-free testsrun: ./gradlew --no-daemon test- # 963 of the 980 registered tests run without a checkpoint. This floor has been raised+ # 1113 of the 1133 registered tests run without a checkpoint. This floor has been raised# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor# that does not bind is not a gate: under 625 the whole confidence-gate family of 26# tests could have vanished from this lane with the gate still green.@@ -83,7 +83,7 @@ jobs:- name: The suite was not vacuousrun: >python3 scripts/check_test_results.py laya-java/build/test-results/test- --min-tests 963 --allow-aborted+ --min-tests 1113 --allow-aborted- name: Build the jarsrun: ./gradlew --no-daemon build -x testdiff --git a/laya-java/CHANGELOG.md b/laya-java/CHANGELOG.mdindex b0d6dad1..cc85c539 100644--- a/laya-java/CHANGELOG.md+++ b/laya-java/CHANGELOG.md@@ -110,6 +110,8 @@ see [Installing](README.md#installing).- `Router`: checkpoint selection with its reason strings, plus a load-and-evict lifecycle withleases, so a checkpoint in use is never closed under a caller.- `Shortlist`: cosine ranking over a caller's embedder, with an LRU cache and numpy's tie order.+- `Shortlist.predictTournament`: elimination rounds over a large choice, a port of+ `predict_tournament`, with no embedder.- `LayaEmail`: `cleanEmailBody` and `emailState`, with the English, Portuguese, Spanish and Frenchmarker sets, and `emailQuestions` re-exported from `Presets`.- `Decisions`: schema-driven decisions — `decide` and `decideBatch` over a JSON schema. Andiff --git a/laya-java/README.md b/laya-java/README.mdindex 6787b608..e6f6361b 100644--- a/laya-java/README.md+++ b/laya-java/README.md@@ -247,6 +247,17 @@ passthrough: the labels come back in order and **the embedder is never called**.Both `Agent` and `Router` implement `Predictor`, so shortlisting works identically against a fixedcheckpoint or a routed one.+Without an embedder, `Shortlist.predictTournament` lets the model narrow the set itself: labels are+answered in near-equal groups of at most 16 (one prediction per round covers every group of every+question), and the winners meet in one final prediction.++```java+Shortlist.Tournament out = Shortlist.predictTournament(agent, state, questions);++out.prediction(); // the final call's answers and usage, over the finalists+out.tournament().get("intent"); // the finalists, the label count and the rounds played+```+## Cleaning an email`laya.email`'s cleaner and state builder. The markers cover English, Portuguese, Spanish anddiff --git a/laya-java/fixtures/tournament.json b/laya-java/fixtures/tournament.jsonnew file mode 100644index 00000000..c12a0065--- /dev/null+++ b/laya-java/fixtures/tournament.json@@ -0,0 +1,5525 @@+{+ "default_group_size": 16,+ "state": "My card was charged twice for the same order last Tuesday.",+ "rank_probe": {+ "": "5951527874303425639",+ "a": "676785770819501618",+ "l0001": "16490875346976899246",+ "注文": "11652523757631009561"+ },+ "splits": [+ {+ "group_size": 2,+ "n": 1,+ "rounds": [],+ "finalists": [+ "l0000"+ ],+ "choice": "l0000"+ },+ {+ "group_size": 2,+ "n": 2,+ "rounds": [],+ "finalists": [+ "l0000",+ "l0001"+ ],+ "choice": "l0001"+ },+ {+ "group_size": 2,+ "n": 3,+ "rounds": [+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0000",+ "l0002"+ ],+ "choice": "l0002"+ },+ {+ "group_size": 2,+ "n": 4,+ "rounds": [+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0001",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 2,+ "n": 5,+ "rounds": [+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0000",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 2,+ "n": 6,+ "rounds": [+ [+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0001",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 2,+ "n": 7,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0002",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 2,+ "n": 8,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0007"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 2,+ "n": 9,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0000",+ "l0008"+ ],+ "choice": "l0008"+ },+ {+ "group_size": 2,+ "n": 10,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0001",+ "l0009"+ ],+ "choice": "l0009"+ },+ {+ "group_size": 2,+ "n": 11,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0002",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 2,+ "n": 12,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 2,+ "n": 13,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 2,+ "n": 14,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 2,+ "n": 15,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 2,+ "n": 16,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 2,+ "n": 17,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0000",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 2,+ "n": 18,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0001",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 2,+ "n": 19,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0002",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 20,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 21,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 22,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 23,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 24,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 25,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0008",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 26,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0009",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 27,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 28,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 29,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 30,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 31,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 32,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 33,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0000",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 34,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0001",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 35,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0002",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 36,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 37,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 38,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 39,+ "rounds": [+ [+ 1,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 2,+ "n": 40,+ "rounds": [+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 2,+ 2,+ 2,+ 2,+ 2+ ],+ [+ 1,+ 2,+ 2+ ],+ [+ 1,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 1,+ "rounds": [],+ "finalists": [+ "l0000"+ ],+ "choice": "l0000"+ },+ {+ "group_size": 3,+ "n": 2,+ "rounds": [],+ "finalists": [+ "l0000",+ "l0001"+ ],+ "choice": "l0001"+ },+ {+ "group_size": 3,+ "n": 3,+ "rounds": [],+ "finalists": [+ "l0000",+ "l0001",+ "l0002"+ ],+ "choice": "l0002"+ },+ {+ "group_size": 3,+ "n": 4,+ "rounds": [+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0001",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 3,+ "n": 5,+ "rounds": [+ [+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0001",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 3,+ "n": 6,+ "rounds": [+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0002",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 3,+ "n": 7,+ "rounds": [+ [+ 2,+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0001",+ "l0003",+ "l0006"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 3,+ "n": 8,+ "rounds": [+ [+ 2,+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0001",+ "l0003",+ "l0007"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 3,+ "n": 9,+ "rounds": [+ [+ 3,+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0002",+ "l0003",+ "l0008"+ ],+ "choice": "l0008"+ },+ {+ "group_size": 3,+ "n": 10,+ "rounds": [+ [+ 2,+ 3,+ 2,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0009"+ ],+ "choice": "l0009"+ },+ {+ "group_size": 3,+ "n": 11,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 3,+ "n": 12,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 3,+ "n": 13,+ "rounds": [+ [+ 2,+ 3,+ 2,+ 3,+ 3+ ],+ [+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 3,+ "n": 14,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 3,+ "n": 15,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 3,+ "n": 16,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 2,+ 3,+ 3+ ],+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 3,+ "n": 17,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 3,+ "n": 18,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0008",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 3,+ "n": 19,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 2,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0009",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 20,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 21,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 22,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 2,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 23,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 24,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 25,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 2,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 3,+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 26,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 3,+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0003",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 27,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 3,+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0008",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 28,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 2,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 2,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 29,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 2,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 30,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 2,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 31,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 2,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 32,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 33,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 34,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 2,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 35,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 36,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 2+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 37,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 2,+ 3,+ 3+ ],+ [+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 38,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 2,+ 3,+ 3+ ],+ [+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 39,+ "rounds": [+ [+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 2,+ 3,+ 3+ ],+ [+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 3,+ "n": 40,+ "rounds": [+ [+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3,+ 2,+ 3,+ 3,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3,+ 3,+ 3,+ 3+ ],+ [+ 2,+ 3+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 1,+ "rounds": [],+ "finalists": [+ "l0000"+ ],+ "choice": "l0000"+ },+ {+ "group_size": 5,+ "n": 2,+ "rounds": [],+ "finalists": [+ "l0000",+ "l0001"+ ],+ "choice": "l0001"+ },+ {+ "group_size": 5,+ "n": 3,+ "rounds": [],+ "finalists": [+ "l0000",+ "l0001",+ "l0002"+ ],+ "choice": "l0002"+ },+ {+ "group_size": 5,+ "n": 4,+ "rounds": [],+ "finalists": [+ "l0000",+ "l0001",+ "l0002",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 5,+ "n": 5,+ "rounds": [],+ "finalists": [+ "l0000",+ "l0001",+ "l0002",+ "l0003",+ "l0004"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 5,+ "n": 6,+ "rounds": [+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0002",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 5,+ "n": 7,+ "rounds": [+ [+ 3,+ 4+ ]+ ],+ "finalists": [+ "l0002",+ "l0003"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 5,+ "n": 8,+ "rounds": [+ [+ 4,+ 4+ ]+ ],+ "finalists": [+ "l0003",+ "l0007"+ ],+ "choice": "l0003"+ },+ {+ "group_size": 5,+ "n": 9,+ "rounds": [+ [+ 4,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0008"+ ],+ "choice": "l0008"+ },+ {+ "group_size": 5,+ "n": 10,+ "rounds": [+ [+ 5,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0009"+ ],+ "choice": "l0009"+ },+ {+ "group_size": 5,+ "n": 11,+ "rounds": [+ [+ 3,+ 4,+ 4+ ]+ ],+ "finalists": [+ "l0002",+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 5,+ "n": 12,+ "rounds": [+ [+ 4,+ 4,+ 4+ ]+ ],+ "finalists": [+ "l0003",+ "l0007",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 5,+ "n": 13,+ "rounds": [+ [+ 4,+ 4,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0007",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 5,+ "n": 14,+ "rounds": [+ [+ 4,+ 5,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0008",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 5,+ "n": 15,+ "rounds": [+ [+ 5,+ 5,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0009",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 5,+ "n": 16,+ "rounds": [+ [+ 4,+ 4,+ 4,+ 4+ ]+ ],+ "finalists": [+ "l0003",+ "l0007",+ "l0010",+ "l0012"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 5,+ "n": 17,+ "rounds": [+ [+ 4,+ 4,+ 4,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0007",+ "l0010",+ "l0012"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 5,+ "n": 18,+ "rounds": [+ [+ 4,+ 5,+ 4,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0008",+ "l0010",+ "l0013"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 5,+ "n": 19,+ "rounds": [+ [+ 4,+ 5,+ 5,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0008",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 20,+ "rounds": [+ [+ 5,+ 5,+ 5,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0009",+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 21,+ "rounds": [+ [+ 4,+ 4,+ 4,+ 4,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0007",+ "l0010",+ "l0012",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 22,+ "rounds": [+ [+ 4,+ 4,+ 5,+ 4,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0007",+ "l0010",+ "l0013",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 23,+ "rounds": [+ [+ 4,+ 5,+ 4,+ 5,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0008",+ "l0010",+ "l0013",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 24,+ "rounds": [+ [+ 4,+ 5,+ 5,+ 5,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0008",+ "l0010",+ "l0018",+ "l0019"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 25,+ "rounds": [+ [+ 5,+ 5,+ 5,+ 5,+ 5+ ]+ ],+ "finalists": [+ "l0003",+ "l0009",+ "l0010",+ "l0018",+ "l0021"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 26,+ "rounds": [+ [+ 4,+ 4,+ 5,+ 4,+ 4,+ 5+ ],+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 27,+ "rounds": [+ [+ 4,+ 5,+ 4,+ 5,+ 4,+ 5+ ],+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 28,+ "rounds": [+ [+ 4,+ 5,+ 5,+ 4,+ 5,+ 5+ ],+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 29,+ "rounds": [+ [+ 4,+ 5,+ 5,+ 5,+ 5,+ 5+ ],+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 30,+ "rounds": [+ [+ 5,+ 5,+ 5,+ 5,+ 5,+ 5+ ],+ [+ 3,+ 3+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 31,+ "rounds": [+ [+ 4,+ 4,+ 5,+ 4,+ 5,+ 4,+ 5+ ],+ [+ 3,+ 4+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 32,+ "rounds": [+ [+ 4,+ 5,+ 4,+ 5,+ 4,+ 5,+ 5+ ],+ [+ 3,+ 4+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 33,+ "rounds": [+ [+ 4,+ 5,+ 5,+ 4,+ 5,+ 5,+ 5+ ],+ [+ 3,+ 4+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 34,+ "rounds": [+ [+ 4,+ 5,+ 5,+ 5,+ 5,+ 5,+ 5+ ],+ [+ 3,+ 4+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 35,+ "rounds": [+ [+ 5,+ 5,+ 5,+ 5,+ 5,+ 5,+ 5+ ],+ [+ 3,+ 4+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 36,+ "rounds": [+ [+ 4,+ 5,+ 4,+ 5,+ 4,+ 5,+ 4,+ 5+ ],+ [+ 4,+ 4+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 37,+ "rounds": [+ [+ 4,+ 5,+ 4,+ 5,+ 5,+ 4,+ 5,+ 5+ ],+ [+ 4,+ 4+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 38,+ "rounds": [+ [+ 4,+ 5,+ 5,+ 5,+ 4,+ 5,+ 5,+ 5+ ],+ [+ 4,+ 4+ ]+ ],+ "finalists": [+ "l0018",+ "l0019"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 39,+ "rounds": [+ [+ 4,+ 5,+ 5,+ 5,+ 5,+ 5,+ 5,+ 5+ ],+ [+ 4,+ 4+ ]+ ],+ "finalists": [+ "l0018",+ "l0019"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 5,+ "n": 40,+ "rounds": [+ [+ 5,+ 5,+ 5,+ 5,+ 5,+ 5,+ 5,+ 5+ ],+ [+ 4,+ 4+ ]+ ],+ "finalists": [+ "l0018",+ "l0038"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 16,+ "n": 1,+ "rounds": [],+ "finalists": [+ "l0000"+ ],+ "choice": "l0000"+ },+ {+ "group_size": 16,+ "n": 15,+ "rounds": [],+ "finalists": [+ "l0000",+ "l0001",+ "l0002",+ "l0003",+ "l0004",+ "l0005",+ "l0006",+ "l0007",+ "l0008",+ "l0009",+ "l0010",+ "l0011",+ "l0012",+ "l0013",+ "l0014"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 16,+ "n": 16,+ "rounds": [],+ "finalists": [+ "l0000",+ "l0001",+ "l0002",+ "l0003",+ "l0004",+ "l0005",+ "l0006",+ "l0007",+ "l0008",+ "l0009",+ "l0010",+ "l0011",+ "l0012",+ "l0013",+ "l0014",+ "l0015"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 16,+ "n": 17,+ "rounds": [+ [+ 8,+ 9+ ]+ ],+ "finalists": [+ "l0003",+ "l0010"+ ],+ "choice": "l0010"+ },+ {+ "group_size": 16,+ "n": 31,+ "rounds": [+ [+ 15,+ 16+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 16,+ "n": 32,+ "rounds": [+ [+ 16,+ 16+ ]+ ],+ "finalists": [+ "l0010",+ "l0018"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 16,+ "n": 33,+ "rounds": [+ [+ 11,+ 11,+ 11+ ]+ ],+ "finalists": [+ "l0010",+ "l0018",+ "l0032"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 16,+ "n": 77,+ "rounds": [+ [+ 15,+ 15,+ 16,+ 15,+ 16+ ]+ ],+ "finalists": [+ "l0010",+ "l0018",+ "l0038",+ "l0054",+ "l0072"+ ],+ "choice": "l0018"+ },+ {+ "group_size": 16,+ "n": 150,+ "rounds": [+ [+ 15,+ 15,+ 15,+ 15,+ 15,+ 15,+ 15,+ 15,+ 15,+ 15+ ]+ ],+ "finalists": [+ "l0010",+ "l0018",+ "l0038",+ "l0054",+ "l0072",+ "l0083",+ "l0102",+ "l0106",+ "l0128",+ "l0146"+ ],+ "choice": "l0146"+ },+ {+ "group_size": 16,+ "n": 255,+ "rounds": [+ [+ 15,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16+ ]+ ],+ "finalists": [+ "l0010",+ "l0018",+ "l0038",+ "l0054",+ "l0078",+ "l0090",+ "l0102",+ "l0124",+ "l0142",+ "l0146",+ "l0160",+ "l0177",+ "l0201",+ "l0222",+ "l0230",+ "l0239"+ ],+ "choice": "l0230"+ },+ {+ "group_size": 16,+ "n": 256,+ "rounds": [+ [+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16+ ]+ ],+ "finalists": [+ "l0010",+ "l0018",+ "l0038",+ "l0054",+ "l0078",+ "l0090",+ "l0102",+ "l0124",+ "l0142",+ "l0146",+ "l0160",+ "l0177",+ "l0201",+ "l0223",+ "l0230",+ "l0252"+ ],+ "choice": "l0230"+ },+ {+ "group_size": 16,+ "n": 257,+ "rounds": [+ [+ 15,+ 15,+ 15,+ 15,+ 15,+ 15,+ 15,+ 15,+ 16,+ 15,+ 15,+ 15,+ 15,+ 15,+ 15,+ 15,+ 16+ ],+ [+ 8,+ 9+ ]+ ],+ "finalists": [+ "l0102",+ "l0230"+ ],+ "choice": "l0230"+ },+ {+ "group_size": 16,+ "n": 300,+ "rounds": [+ [+ 15,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16+ ],+ [+ 9,+ 10+ ]+ ],+ "finalists": [+ "l0140",+ "l0230"+ ],+ "choice": "l0230"+ },+ {+ "group_size": 16,+ "n": 1000,+ "rounds": [+ [+ 15,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 15,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16,+ 16+ ],+ [+ 15,+ 16,+ 16,+ 16+ ]+ ],+ "finalists": [+ "l0230",+ "l0450",+ "l0494",+ "l0982"+ ],+ "choice": "l0450"+ }+ ],+ "cases": [+ {+ "name": "default-group-77",+ "group_size": null,+ "questions": {+ "q": {+ "type": "choice",+ "instructions": "pick",+ "criteria": [+ "l000",+ "l001",+ "l002",+ "l003",+ "l004",+ "l005",+ "l006",+ "l007",+ "l008",+ "l009",+ "l010",+ "l011",+ "l012",+ "l013",+ "l014",+ "l015",+ "l016",+ "l017",+ "l018",+ "l019",+ "l020",+ "l021",+ "l022",+ "l023",+ "l024",+ "l025",+ "l026",+ "l027",+ "l028",+ "l029",+ "l030",+ "l031",+ "l032",+ "l033",+ "l034",+ "l035",+ "l036",+ "l037",+ "l038",+ "l039",+ "l040",+ "l041",+ "l042",+ "l043",+ "l044",+ "l045",+ "l046",+ "l047",+ "l048",+ "l049",+ "l050",+ "l051",+ "l052",+ "l053",+ "l054",+ "l055",+ "l056",+ "l057",+ "l058",+ "l059",+ "l060",+ "l061",+ "l062",+ "l063",+ "l064",+ "l065",+ "l066",+ "l067",+ "l068",+ "l069",+ "l070",+ "l071",+ "l072",+ "l073",+ "l074",+ "l075",+ "l076"+ ]+ }+ },+ "calls": [+ {+ "0": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l000",+ "l001",+ "l002",+ "l003",+ "l004",+ "l005",+ "l006",+ "l007",+ "l008",+ "l009",+ "l010",+ "l011",+ "l012",+ "l013",+ "l014"+ ]+ },+ "1": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l015",+ "l016",+ "l017",+ "l018",+ "l019",+ "l020",+ "l021",+ "l022",+ "l023",+ "l024",+ "l025",+ "l026",+ "l027",+ "l028",+ "l029"+ ]+ },+ "2": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l030",+ "l031",+ "l032",+ "l033",+ "l034",+ "l035",+ "l036",+ "l037",+ "l038",+ "l039",+ "l040",+ "l041",+ "l042",+ "l043",+ "l044",+ "l045"+ ]+ },+ "3": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l046",+ "l047",+ "l048",+ "l049",+ "l050",+ "l051",+ "l052",+ "l053",+ "l054",+ "l055",+ "l056",+ "l057",+ "l058",+ "l059",+ "l060"+ ]+ },+ "4": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l061",+ "l062",+ "l063",+ "l064",+ "l065",+ "l066",+ "l067",+ "l068",+ "l069",+ "l070",+ "l071",+ "l072",+ "l073",+ "l074",+ "l075",+ "l076"+ ]+ }+ },+ {+ "q": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l014",+ "l023",+ "l030",+ "l060",+ "l074"+ ]+ }+ }+ ],+ "model": "stub-2",+ "answers": {+ "q": "l030"+ },+ "tournament": {+ "q": {+ "labels": [+ "l014",+ "l023",+ "l030",+ "l060",+ "l074"+ ],+ "n": 77,+ "rounds": 1+ }+ },+ "caller_questions_unmutated": true+ },+ {+ "name": "mixed-group-8",+ "group_size": 8,+ "questions": {+ "big": {+ "type": "choice",+ "instructions": "pick",+ "criteria": {+ "l00": "about l00",+ "l01": "about l01",+ "l02": "about l02",+ "l03": "about l03",+ "l04": "about l04",+ "l05": "about l05",+ "l06": "about l06",+ "l07": "about l07",+ "l08": "about l08",+ "l09": "about l09",+ "l10": "about l10",+ "l11": "about l11",+ "l12": "about l12",+ "l13": "about l13",+ "l14": "about l14",+ "l15": "about l15",+ "l16": "about l16",+ "l17": "about l17",+ "l18": "about l18",+ "l19": "about l19"+ }+ },+ "small": {+ "type": "choice",+ "instructions": "pick",+ "criteria": [+ "x",+ "y"+ ]+ },+ "level": {+ "type": "score",+ "instructions": "rate",+ "criteria": [+ "low",+ "high"+ ]+ },+ "urgent": {+ "type": "noul",+ "instructions": "Is this urgent?"+ }+ },+ "calls": [+ {+ "0": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l00",+ "l01",+ "l02",+ "l03",+ "l04",+ "l05"+ ],+ "criteria": {+ "l00": "about l00",+ "l01": "about l01",+ "l02": "about l02",+ "l03": "about l03",+ "l04": "about l04",+ "l05": "about l05"+ }+ },+ "1": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l06",+ "l07",+ "l08",+ "l09",+ "l10",+ "l11",+ "l12"+ ],+ "criteria": {+ "l06": "about l06",+ "l07": "about l07",+ "l08": "about l08",+ "l09": "about l09",+ "l10": "about l10",+ "l11": "about l11",+ "l12": "about l12"+ }+ },+ "2": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l13",+ "l14",+ "l15",+ "l16",+ "l17",+ "l18",+ "l19"+ ],+ "criteria": {+ "l13": "about l13",+ "l14": "about l14",+ "l15": "about l15",+ "l16": "about l16",+ "l17": "about l17",+ "l18": "about l18",+ "l19": "about l19"+ }+ }+ },+ {+ "big": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "l03",+ "l10",+ "l13"+ ],+ "criteria": {+ "l03": "about l03",+ "l10": "about l10",+ "l13": "about l13"+ }+ },+ "small": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "x",+ "y"+ ]+ },+ "level": {+ "type": "score",+ "instructions": "rate"+ },+ "urgent": {+ "type": "noul",+ "instructions": "Is this urgent?"+ }+ }+ ],+ "model": "stub-2",+ "answers": {+ "big": "l10",+ "small": "y"+ },+ "tournament": {+ "big": {+ "labels": [+ "l03",+ "l10",+ "l13"+ ],+ "n": 20,+ "rounds": 1+ },+ "small": {+ "labels": [+ "x",+ "y"+ ],+ "n": 2,+ "rounds": 0+ }+ },+ "caller_questions_unmutated": true+ },+ {+ "name": "nothing-to-narrow",+ "group_size": null,+ "questions": {+ "small": {+ "type": "choice",+ "instructions": "pick",+ "criteria": [+ "x",+ "y"+ ]+ },+ "level": {+ "type": "score",+ "instructions": "rate",+ "criteria": [+ "low",+ "high"+ ]+ }+ },+ "calls": [+ {+ "small": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "x",+ "y"+ ]+ },+ "level": {+ "type": "score",+ "instructions": "rate"+ }+ }+ ],+ "model": "stub-1",+ "answers": {+ "small": "y"+ },+ "tournament": {+ "small": {+ "labels": [+ "x",+ "y"+ ],+ "n": 2,+ "rounds": 0+ }+ },+ "caller_questions_unmutated": true+ },+ {+ "name": "two-choices-different-rounds",+ "group_size": null,+ "questions": {+ "wide": {+ "type": "choice",+ "instructions": "Which one?",+ "criteria": [+ "w000",+ "w001",+ "w002",+ "w003",+ "w004",+ "w005",+ "w006",+ "w007",+ "w008",+ "w009",+ "w010",+ "w011",+ "w012",+ "w013",+ "w014",+ "w015",+ "w016",+ "w017",+ "w018",+ "w019",+ "w020",+ "w021",+ "w022",+ "w023",+ "w024",+ "w025",+ "w026",+ "w027",+ "w028",+ "w029",+ "w030",+ "w031",+ "w032",+ "w033",+ "w034",+ "w035",+ "w036",+ "w037",+ "w038",+ "w039",+ "w040",+ "w041",+ "w042",+ "w043",+ "w044",+ "w045",+ "w046",+ "w047",+ "w048",+ "w049",+ "w050",+ "w051",+ "w052",+ "w053",+ "w054",+ "w055",+ "w056",+ "w057",+ "w058",+ "w059",+ "w060",+ "w061",+ "w062",+ "w063",+ "w064",+ "w065",+ "w066",+ "w067",+ "w068",+ "w069",+ "w070",+ "w071",+ "w072",+ "w073",+ "w074",+ "w075",+ "w076",+ "w077",+ "w078",+ "w079",+ "w080",+ "w081",+ "w082",+ "w083",+ "w084",+ "w085",+ "w086",+ "w087",+ "w088",+ "w089",+ "w090",+ "w091",+ "w092",+ "w093",+ "w094",+ "w095",+ "w096",+ "w097",+ "w098",+ "w099",+ "w100",+ "w101",+ "w102",+ "w103",+ "w104",+ "w105",+ "w106",+ "w107",+ "w108",+ "w109",+ "w110",+ "w111",+ "w112",+ "w113",+ "w114",+ "w115",+ "w116",+ "w117",+ "w118",+ "w119",+ "w120",+ "w121",+ "w122",+ "w123",+ "w124",+ "w125",+ "w126",+ "w127",+ "w128",+ "w129",+ "w130",+ "w131",+ "w132",+ "w133",+ "w134",+ "w135",+ "w136",+ "w137",+ "w138",+ "w139",+ "w140",+ "w141",+ "w142",+ "w143",+ "w144",+ "w145",+ "w146",+ "w147",+ "w148",+ "w149",+ "w150",+ "w151",+ "w152",+ "w153",+ "w154",+ "w155",+ "w156",+ "w157",+ "w158",+ "w159",+ "w160",+ "w161",+ "w162",+ "w163",+ "w164",+ "w165",+ "w166",+ "w167",+ "w168",+ "w169",+ "w170",+ "w171",+ "w172",+ "w173",+ "w174",+ "w175",+ "w176",+ "w177",+ "w178",+ "w179",+ "w180",+ "w181",+ "w182",+ "w183",+ "w184",+ "w185",+ "w186",+ "w187",+ "w188",+ "w189",+ "w190",+ "w191",+ "w192",+ "w193",+ "w194",+ "w195",+ "w196",+ "w197",+ "w198",+ "w199",+ "w200",+ "w201",+ "w202",+ "w203",+ "w204",+ "w205",+ "w206",+ "w207",+ "w208",+ "w209",+ "w210",+ "w211",+ "w212",+ "w213",+ "w214",+ "w215",+ "w216",+ "w217",+ "w218",+ "w219",+ "w220",+ "w221",+ "w222",+ "w223",+ "w224",+ "w225",+ "w226",+ "w227",+ "w228",+ "w229",+ "w230",+ "w231",+ "w232",+ "w233",+ "w234",+ "w235",+ "w236",+ "w237",+ "w238",+ "w239",+ "w240",+ "w241",+ "w242",+ "w243",+ "w244",+ "w245",+ "w246",+ "w247",+ "w248",+ "w249",+ "w250",+ "w251",+ "w252",+ "w253",+ "w254",+ "w255",+ "w256",+ "w257",+ "w258",+ "w259",+ "w260",+ "w261",+ "w262",+ "w263",+ "w264",+ "w265",+ "w266",+ "w267",+ "w268",+ "w269",+ "w270",+ "w271",+ "w272",+ "w273",+ "w274",+ "w275",+ "w276",+ "w277",+ "w278",+ "w279",+ "w280",+ "w281",+ "w282",+ "w283",+ "w284",+ "w285",+ "w286",+ "w287",+ "w288",+ "w289",+ "w290",+ "w291",+ "w292",+ "w293",+ "w294",+ "w295",+ "w296",+ "w297",+ "w298",+ "w299"+ ]+ },+ "narrow": {+ "type": "choice",+ "instructions": "Which other?",+ "criteria": [+ "n00",+ "n01",+ "n02",+ "n03",+ "n04",+ "n05",+ "n06",+ "n07",+ "n08",+ "n09",+ "n10",+ "n11",+ "n12",+ "n13",+ "n14",+ "n15",+ "n16",+ "n17",+ "n18",+ "n19",+ "n20",+ "n21",+ "n22",+ "n23",+ "n24",+ "n25",+ "n26",+ "n27",+ "n28",+ "n29",+ "n30",+ "n31",+ "n32",+ "n33",+ "n34",+ "n35",+ "n36",+ "n37",+ "n38",+ "n39"+ ]+ }+ },+ "calls": [+ {+ "0": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w000",+ "w001",+ "w002",+ "w003",+ "w004",+ "w005",+ "w006",+ "w007",+ "w008",+ "w009",+ "w010",+ "w011",+ "w012",+ "w013",+ "w014"+ ]+ },+ "1": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w015",+ "w016",+ "w017",+ "w018",+ "w019",+ "w020",+ "w021",+ "w022",+ "w023",+ "w024",+ "w025",+ "w026",+ "w027",+ "w028",+ "w029",+ "w030"+ ]+ },+ "2": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w031",+ "w032",+ "w033",+ "w034",+ "w035",+ "w036",+ "w037",+ "w038",+ "w039",+ "w040",+ "w041",+ "w042",+ "w043",+ "w044",+ "w045",+ "w046"+ ]+ },+ "3": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w047",+ "w048",+ "w049",+ "w050",+ "w051",+ "w052",+ "w053",+ "w054",+ "w055",+ "w056",+ "w057",+ "w058",+ "w059",+ "w060",+ "w061",+ "w062"+ ]+ },+ "4": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w063",+ "w064",+ "w065",+ "w066",+ "w067",+ "w068",+ "w069",+ "w070",+ "w071",+ "w072",+ "w073",+ "w074",+ "w075",+ "w076",+ "w077"+ ]+ },+ "5": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w078",+ "w079",+ "w080",+ "w081",+ "w082",+ "w083",+ "w084",+ "w085",+ "w086",+ "w087",+ "w088",+ "w089",+ "w090",+ "w091",+ "w092",+ "w093"+ ]+ },+ "6": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w094",+ "w095",+ "w096",+ "w097",+ "w098",+ "w099",+ "w100",+ "w101",+ "w102",+ "w103",+ "w104",+ "w105",+ "w106",+ "w107",+ "w108",+ "w109"+ ]+ },+ "7": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w110",+ "w111",+ "w112",+ "w113",+ "w114",+ "w115",+ "w116",+ "w117",+ "w118",+ "w119",+ "w120",+ "w121",+ "w122",+ "w123",+ "w124",+ "w125"+ ]+ },+ "8": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w126",+ "w127",+ "w128",+ "w129",+ "w130",+ "w131",+ "w132",+ "w133",+ "w134",+ "w135",+ "w136",+ "w137",+ "w138",+ "w139",+ "w140",+ "w141"+ ]+ },+ "9": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w142",+ "w143",+ "w144",+ "w145",+ "w146",+ "w147",+ "w148",+ "w149",+ "w150",+ "w151",+ "w152",+ "w153",+ "w154",+ "w155",+ "w156"+ ]+ },+ "10": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w157",+ "w158",+ "w159",+ "w160",+ "w161",+ "w162",+ "w163",+ "w164",+ "w165",+ "w166",+ "w167",+ "w168",+ "w169",+ "w170",+ "w171",+ "w172"+ ]+ },+ "11": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w173",+ "w174",+ "w175",+ "w176",+ "w177",+ "w178",+ "w179",+ "w180",+ "w181",+ "w182",+ "w183",+ "w184",+ "w185",+ "w186",+ "w187",+ "w188"+ ]+ },+ "12": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w189",+ "w190",+ "w191",+ "w192",+ "w193",+ "w194",+ "w195",+ "w196",+ "w197",+ "w198",+ "w199",+ "w200",+ "w201",+ "w202",+ "w203",+ "w204"+ ]+ },+ "13": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w205",+ "w206",+ "w207",+ "w208",+ "w209",+ "w210",+ "w211",+ "w212",+ "w213",+ "w214",+ "w215",+ "w216",+ "w217",+ "w218",+ "w219",+ "w220"+ ]+ },+ "14": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w221",+ "w222",+ "w223",+ "w224",+ "w225",+ "w226",+ "w227",+ "w228",+ "w229",+ "w230",+ "w231",+ "w232",+ "w233",+ "w234",+ "w235"+ ]+ },+ "15": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w236",+ "w237",+ "w238",+ "w239",+ "w240",+ "w241",+ "w242",+ "w243",+ "w244",+ "w245",+ "w246",+ "w247",+ "w248",+ "w249",+ "w250",+ "w251"+ ]+ },+ "16": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w252",+ "w253",+ "w254",+ "w255",+ "w256",+ "w257",+ "w258",+ "w259",+ "w260",+ "w261",+ "w262",+ "w263",+ "w264",+ "w265",+ "w266",+ "w267"+ ]+ },+ "17": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w268",+ "w269",+ "w270",+ "w271",+ "w272",+ "w273",+ "w274",+ "w275",+ "w276",+ "w277",+ "w278",+ "w279",+ "w280",+ "w281",+ "w282",+ "w283"+ ]+ },+ "18": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w284",+ "w285",+ "w286",+ "w287",+ "w288",+ "w289",+ "w290",+ "w291",+ "w292",+ "w293",+ "w294",+ "w295",+ "w296",+ "w297",+ "w298",+ "w299"+ ]+ },+ "19": {+ "type": "choice",+ "instructions": "Which other?",+ "labels": [+ "n00",+ "n01",+ "n02",+ "n03",+ "n04",+ "n05",+ "n06",+ "n07",+ "n08",+ "n09",+ "n10",+ "n11",+ "n12"+ ]+ },+ "20": {+ "type": "choice",+ "instructions": "Which other?",+ "labels": [+ "n13",+ "n14",+ "n15",+ "n16",+ "n17",+ "n18",+ "n19",+ "n20",+ "n21",+ "n22",+ "n23",+ "n24",+ "n25"+ ]+ },+ "21": {+ "type": "choice",+ "instructions": "Which other?",+ "labels": [+ "n26",+ "n27",+ "n28",+ "n29",+ "n30",+ "n31",+ "n32",+ "n33",+ "n34",+ "n35",+ "n36",+ "n37",+ "n38",+ "n39"+ ]+ }+ },+ {+ "0": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w000",+ "w017",+ "w039",+ "w048",+ "w063",+ "w079",+ "w099",+ "w123",+ "w130"+ ]+ },+ "1": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w152",+ "w170",+ "w178",+ "w198",+ "w220",+ "w228",+ "w246",+ "w260",+ "w268",+ "w288"+ ]+ }+ },+ {+ "wide": {+ "type": "choice",+ "instructions": "Which one?",+ "labels": [+ "w000",+ "w178"+ ]+ },+ "narrow": {+ "type": "choice",+ "instructions": "Which other?",+ "labels": [+ "n12",+ "n23",+ "n38"+ ]+ }+ }+ ],+ "model": "stub-3",+ "answers": {+ "wide": "w000",+ "narrow": "n38"+ },+ "tournament": {+ "wide": {+ "labels": [+ "w000",+ "w178"+ ],+ "n": 300,+ "rounds": 2+ },+ "narrow": {+ "labels": [+ "n12",+ "n23",+ "n38"+ ],+ "n": 40,+ "rounds": 1+ }+ },+ "caller_questions_unmutated": true+ },+ {+ "name": "described-criteria",+ "group_size": 3,+ "questions": {+ "intent": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "criteria": {+ "card_arrival": null,+ "card_delivery_estimate": "",+ "card_not_working": 2,+ "cash_withdrawal_charge": "cash withdrawal charge",+ "declined_card_payment": null,+ "direct_debit_payment_not_recognised": "",+ "exchange_rate": 6,+ "failed_transfer": "failed transfer",+ "lost_or_stolen_card": null,+ "pending_card_payment": "",+ "refund_not_showing_up": 10,+ "request_refund": "request refund",+ "reverted_card_payment": null,+ "top_up_failed": "",+ "transfer_fee_charged": 14,+ "verify_my_identity": "verify my identity",+ "wrong_amount_of_cash_received": null+ }+ }+ },+ "calls": [+ {+ "0": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "card_arrival",+ "card_delivery_estimate"+ ],+ "criteria": {+ "card_arrival": null,+ "card_delivery_estimate": ""+ }+ },+ "1": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "card_not_working",+ "cash_withdrawal_charge",+ "declined_card_payment"+ ],+ "criteria": {+ "card_not_working": 2,+ "cash_withdrawal_charge": "cash withdrawal charge",+ "declined_card_payment": null+ }+ },+ "2": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "direct_debit_payment_not_recognised",+ "exchange_rate",+ "failed_transfer"+ ],+ "criteria": {+ "direct_debit_payment_not_recognised": "",+ "exchange_rate": 6,+ "failed_transfer": "failed transfer"+ }+ },+ "3": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "lost_or_stolen_card",+ "pending_card_payment",+ "refund_not_showing_up"+ ],+ "criteria": {+ "lost_or_stolen_card": null,+ "pending_card_payment": "",+ "refund_not_showing_up": 10+ }+ },+ "4": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "request_refund",+ "reverted_card_payment",+ "top_up_failed"+ ],+ "criteria": {+ "request_refund": "request refund",+ "reverted_card_payment": null,+ "top_up_failed": ""+ }+ },+ "5": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "transfer_fee_charged",+ "verify_my_identity",+ "wrong_amount_of_cash_received"+ ],+ "criteria": {+ "transfer_fee_charged": 14,+ "verify_my_identity": "verify my identity",+ "wrong_amount_of_cash_received": null+ }+ }+ },+ {+ "0": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "card_delivery_estimate",+ "card_not_working",+ "direct_debit_payment_not_recognised"+ ],+ "criteria": {+ "card_delivery_estimate": "",+ "card_not_working": 2,+ "direct_debit_payment_not_recognised": ""+ }+ },+ "1": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "refund_not_showing_up",+ "top_up_failed",+ "wrong_amount_of_cash_received"+ ],+ "criteria": {+ "refund_not_showing_up": 10,+ "top_up_failed": "",+ "wrong_amount_of_cash_received": null+ }+ }+ },+ {+ "intent": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "direct_debit_payment_not_recognised",+ "top_up_failed"+ ],+ "criteria": {+ "direct_debit_payment_not_recognised": "",+ "top_up_failed": ""+ }+ }+ }+ ],+ "model": "stub-3",+ "answers": {+ "intent": "top_up_failed"+ },+ "tournament": {+ "intent": {+ "labels": [+ "direct_debit_payment_not_recognised",+ "top_up_failed"+ ],+ "n": 17,+ "rounds": 2+ }+ },+ "caller_questions_unmutated": true+ },+ {+ "name": "group-of-one",+ "group_size": 2,+ "questions": {+ "q": {+ "type": "choice",+ "instructions": "pick",+ "criteria": [+ "e",+ "d",+ "c",+ "b",+ "a"+ ]+ }+ },+ "calls": [+ {+ "0": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "e"+ ]+ },+ "1": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "d",+ "c"+ ]+ },+ "2": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "b",+ "a"+ ]+ }+ },+ {+ "0": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "e"+ ]+ },+ "1": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "d",+ "b"+ ]+ }+ },+ {+ "q": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "e",+ "d"+ ]+ }+ }+ ],+ "model": "stub-3",+ "answers": {+ "q": "e"+ },+ "tournament": {+ "q": {+ "labels": [+ "e",+ "d"+ ],+ "n": 5,+ "rounds": 2+ }+ },+ "caller_questions_unmutated": true+ },+ {+ "name": "criteria-order-not-sorted",+ "group_size": 4,+ "questions": {+ "q": {+ "type": "choice",+ "instructions": "pick",+ "criteria": [+ "wrong_amount_of_cash_received",+ "verify_my_identity",+ "transfer_fee_charged",+ "top_up_failed",+ "reverted_card_payment",+ "request_refund",+ "refund_not_showing_up",+ "pending_card_payment",+ "lost_or_stolen_card",+ "failed_transfer",+ "exchange_rate",+ "direct_debit_payment_not_recognised",+ "declined_card_payment",+ "cash_withdrawal_charge",+ "card_not_working",+ "card_delivery_estimate",+ "card_arrival"+ ]+ }+ },+ "calls": [+ {+ "0": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "wrong_amount_of_cash_received",+ "verify_my_identity",+ "transfer_fee_charged"+ ]+ },+ "1": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "top_up_failed",+ "reverted_card_payment",+ "request_refund"+ ]+ },+ "2": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "refund_not_showing_up",+ "pending_card_payment",+ "lost_or_stolen_card",+ "failed_transfer"+ ]+ },+ "3": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "exchange_rate",+ "direct_debit_payment_not_recognised",+ "declined_card_payment"+ ]+ },+ "4": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "cash_withdrawal_charge",+ "card_not_working",+ "card_delivery_estimate",+ "card_arrival"+ ]+ }+ },+ {+ "0": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "wrong_amount_of_cash_received",+ "top_up_failed"+ ]+ },+ "1": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "refund_not_showing_up",+ "direct_debit_payment_not_recognised",+ "card_not_working"+ ]+ }+ },+ {+ "q": {+ "type": "choice",+ "instructions": "pick",+ "labels": [+ "top_up_failed",+ "direct_debit_payment_not_recognised"+ ]+ }+ }+ ],+ "model": "stub-3",+ "answers": {+ "q": "top_up_failed"+ },+ "tournament": {+ "q": {+ "labels": [+ "top_up_failed",+ "direct_debit_payment_not_recognised"+ ],+ "n": 17,+ "rounds": 2+ }+ },+ "caller_questions_unmutated": true+ }+ ],+ "refusals": [+ {+ "group_size": 1,+ "error": "ValueError",+ "message": "group_size must be an integer of at least 2, got 1"+ },+ {+ "group_size": 0,+ "error": "ValueError",+ "message": "group_size must be an integer of at least 2, got 0"+ },+ {+ "group_size": -3,+ "error": "ValueError",+ "message": "group_size must be an integer of at least 2, got -3"+ }+ ]+}diff --git a/laya-java/fixtures/tournament_golden.json b/laya-java/fixtures/tournament_golden.jsonnew file mode 100644index 00000000..69cdc1d5--- /dev/null+++ b/laya-java/fixtures/tournament_golden.json@@ -0,0 +1,509 @@+{+ "checkpoint": "multilingual",+ "graph": "laya.onnx",+ "cases": [+ {+ "name": "banking-default-group",+ "state": "My new card still has not arrived after two weeks.",+ "group_size": null,+ "questions": {+ "intent": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "criteria": [+ "activate_my_card",+ "age_limit",+ "apple_pay_or_google_pay",+ "atm_support",+ "automatic_top_up",+ "balance_not_updated_after_bank_transfer",+ "beneficiary_not_allowed",+ "cancel_transfer",+ "card_about_to_expire",+ "card_acceptance",+ "card_arrival",+ "card_delivery_estimate",+ "card_linking",+ "card_not_working",+ "card_payment_fee_charged",+ "card_payment_not_recognised",+ "card_swallowed",+ "cash_withdrawal_charge",+ "change_pin",+ "compromised_card",+ "contactless_not_working",+ "country_support",+ "declined_card_payment",+ "declined_cash_withdrawal",+ "declined_transfer",+ "edit_personal_details",+ "exchange_charge",+ "exchange_rate",+ "failed_transfer",+ "fiat_currency_support",+ "get_physical_card",+ "lost_or_stolen_card",+ "lost_or_stolen_phone",+ "order_physical_card",+ "passcode_forgotten",+ "pending_card_payment",+ "refund_not_showing_up",+ "request_refund",+ "terminate_account",+ "top_up_failed"+ ]+ }+ },+ "calls": [+ {+ "asked": {+ "0": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "activate_my_card",+ "age_limit",+ "apple_pay_or_google_pay",+ "atm_support",+ "automatic_top_up",+ "balance_not_updated_after_bank_transfer",+ "beneficiary_not_allowed",+ "cancel_transfer",+ "card_about_to_expire",+ "card_acceptance",+ "card_arrival",+ "card_delivery_estimate",+ "card_linking"+ ]+ },+ "1": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "card_not_working",+ "card_payment_fee_charged",+ "card_payment_not_recognised",+ "card_swallowed",+ "cash_withdrawal_charge",+ "change_pin",+ "compromised_card",+ "contactless_not_working",+ "country_support",+ "declined_card_payment",+ "declined_cash_withdrawal",+ "declined_transfer",+ "edit_personal_details"+ ]+ },+ "2": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "exchange_charge",+ "exchange_rate",+ "failed_transfer",+ "fiat_currency_support",+ "get_physical_card",+ "lost_or_stolen_card",+ "lost_or_stolen_phone",+ "order_physical_card",+ "passcode_forgotten",+ "pending_card_payment",+ "refund_not_showing_up",+ "request_refund",+ "terminate_account",+ "top_up_failed"+ ]+ }+ },+ "answers": {+ "0": {+ "type": "choice",+ "choice": "card_about_to_expire",+ "probabilities": {+ "activate_my_card": 0.0049,+ "age_limit": 0.0001,+ "apple_pay_or_google_pay": 0.0001,+ "atm_support": 0.0001,+ "automatic_top_up": 0.0002,+ "balance_not_updated_after_bank_transfer": 0.0004,+ "beneficiary_not_allowed": 0.0001,+ "cancel_transfer": 0.0001,+ "card_about_to_expire": 0.8734,+ "card_acceptance": 0.0036,+ "card_arrival": 0.1124,+ "card_delivery_estimate": 0.0041,+ "card_linking": 0.0006+ },+ "confidence": 0.8264,+ "answer_confidence": 0.8734,+ "action": {+ "act_probability": 1.0+ }+ },+ "1": {+ "type": "choice",+ "choice": "card_not_working",+ "probabilities": {+ "card_not_working": 0.9967,+ "card_payment_fee_charged": 0.0,+ "card_payment_not_recognised": 0.0001,+ "card_swallowed": 0.0009,+ "cash_withdrawal_charge": 0.0,+ "change_pin": 0.0,+ "compromised_card": 0.0016,+ "contactless_not_working": 0.0005,+ "country_support": 0.0,+ "declined_card_payment": 0.0,+ "declined_cash_withdrawal": 0.0,+ "declined_transfer": 0.0,+ "edit_personal_details": 0.0+ },+ "confidence": 0.9898,+ "answer_confidence": 0.9967,+ "action": {+ "act_probability": 1.0+ }+ },+ "2": {+ "type": "choice",+ "choice": "lost_or_stolen_card",+ "probabilities": {+ "exchange_charge": 0.0001,+ "exchange_rate": 0.0002,+ "failed_transfer": 0.0929,+ "fiat_currency_support": 0.0003,+ "get_physical_card": 0.0002,+ "lost_or_stolen_card": 0.7004,+ "lost_or_stolen_phone": 0.017,+ "order_physical_card": 0.003,+ "passcode_forgotten": 0.016,+ "pending_card_payment": 0.1542,+ "refund_not_showing_up": 0.0015,+ "request_refund": 0.0001,+ "terminate_account": 0.0027,+ "top_up_failed": 0.0116+ },+ "confidence": 0.6228,+ "answer_confidence": 0.7004,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 303,+ "output_tokens": 0,+ "state_tokens": 11,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "asked": {+ "intent": {+ "type": "choice",+ "instructions": "What does the customer want?",+ "labels": [+ "card_about_to_expire",+ "card_not_working",+ "lost_or_stolen_card"+ ]+ }+ },+ "answers": {+ "intent": {+ "type": "choice",+ "choice": "card_about_to_expire",+ "probabilities": {+ "card_about_to_expire": 0.9919,+ "card_not_working": 0.0032,+ "lost_or_stolen_card": 0.0049+ },+ "confidence": 0.9523,+ "answer_confidence": 0.9919,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 46,+ "output_tokens": 0,+ "state_tokens": 11,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "model": "laya-rl-agent-onnx",+ "tournament": {+ "intent": {+ "labels": [+ "card_about_to_expire",+ "card_not_working",+ "lost_or_stolen_card"+ ],+ "n": 40,+ "rounds": 1+ }+ }+ },+ {+ "name": "mixed-group-8",+ "state": "We were billed twice for March and want a refund today.",+ "group_size": 8,+ "questions": {+ "dept": {+ "type": "choice",+ "instructions": "Which team owns this?",+ "criteria": {+ "billing": "invoices and charges",+ "refunds": "money sent back",+ "fraud": "unauthorised activity",+ "cards": "card problems",+ "transfers": "bank transfers",+ "loans": "credit and loans",+ "savings": "savings products",+ "kyc": "identity checks",+ "support": "general help",+ "legal": "complaints and law",+ "sales": "new products",+ "tech": "app bugs"+ }+ },+ "urgent": {+ "type": "noul",+ "instructions": "This needs a human today."+ }+ },+ "calls": [+ {+ "asked": {+ "0": {+ "type": "choice",+ "instructions": "Which team owns this?",+ "labels": [+ "billing",+ "refunds",+ "fraud",+ "cards",+ "transfers",+ "loans"+ ],+ "criteria": {+ "billing": "invoices and charges",+ "refunds": "money sent back",+ "fraud": "unauthorised activity",+ "cards": "card problems",+ "transfers": "bank transfers",+ "loans": "credit and loans"+ }+ },+ "1": {+ "type": "choice",+ "instructions": "Which team owns this?",+ "labels": [+ "savings",+ "kyc",+ "support",+ "legal",+ "sales",+ "tech"+ ],+ "criteria": {+ "savings": "savings products",+ "kyc": "identity checks",+ "support": "general help",+ "legal": "complaints and law",+ "sales": "new products",+ "tech": "app bugs"+ }+ }+ },+ "answers": {+ "0": {+ "type": "choice",+ "choice": "refunds",+ "probabilities": {+ "billing": 0.0011,+ "refunds": 0.9988,+ "fraud": 0.0,+ "cards": 0.0,+ "transfers": 0.0,+ "loans": 0.0+ },+ "confidence": 0.9949,+ "answer_confidence": 0.9988,+ "action": {+ "act_probability": 1.0+ }+ },+ "1": {+ "type": "choice",+ "choice": "sales",+ "probabilities": {+ "savings": 0.3114,+ "kyc": 0.1803,+ "support": 0.0423,+ "legal": 0.0523,+ "sales": 0.3787,+ "tech": 0.0351+ },+ "confidence": 0.1933,+ "answer_confidence": 0.3787,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 113,+ "output_tokens": 0,+ "state_tokens": 12,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ },+ {+ "asked": {+ "dept": {+ "type": "choice",+ "instructions": "Which team owns this?",+ "labels": [+ "refunds",+ "sales"+ ],+ "criteria": {+ "refunds": "money sent back",+ "sales": "new products"+ }+ },+ "urgent": {+ "type": "noul",+ "instructions": "This needs a human today."+ }+ },+ "answers": {+ "dept": {+ "type": "choice",+ "choice": "refunds",+ "probabilities": {+ "refunds": 1.0,+ "sales": 0.0+ },+ "confidence": 0.9997,+ "answer_confidence": 1.0,+ "action": {+ "act_probability": 1.0+ }+ },+ "urgent": {+ "type": "noul",+ "noul": 0.0283,+ "confidence": 0.9717,+ "answer_confidence": 0.9717,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 79,+ "output_tokens": 0,+ "state_tokens": 12,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "model": "laya-rl-agent-onnx",+ "tournament": {+ "dept": {+ "labels": [+ "refunds",+ "sales"+ ],+ "n": 12,+ "rounds": 1+ }+ }+ },+ {+ "name": "nothing-to-narrow",+ "state": "The app crashes when I open statements.",+ "group_size": null,+ "questions": {+ "intent": {+ "type": "choice",+ "instructions": "What is wrong?",+ "criteria": {+ "bug": "the app fails",+ "billing": "a charge",+ "other": "else"+ }+ }+ },+ "calls": [+ {+ "asked": {+ "intent": {+ "type": "choice",+ "instructions": "What is wrong?",+ "labels": [+ "bug",+ "billing",+ "other"+ ],+ "criteria": {+ "bug": "the app fails",+ "billing": "a charge",+ "other": "else"+ }+ }+ },+ "answers": {+ "intent": {+ "type": "choice",+ "choice": "bug",+ "probabilities": {+ "bug": 0.9925,+ "billing": 0.0015,+ "other": 0.006+ },+ "confidence": 0.9563,+ "answer_confidence": 0.9925,+ "action": {+ "act_probability": 1.0+ }+ }+ },+ "usage": {+ "input_tokens": 34,+ "output_tokens": 0,+ "state_tokens": 8,+ "state_tokens_dropped": 0,+ "truncated": false,+ "truncated_questions": []+ }+ }+ ],+ "model": "laya-rl-agent-onnx",+ "tournament": {+ "intent": {+ "labels": [+ "bug",+ "billing",+ "other"+ ],+ "n": 3,+ "rounds": 0+ }+ }+ }+ ]+}diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Shortlist.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Shortlist.javaindex f4d1ee8c..670bf6e4 100644--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Shortlist.java+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Shortlist.java@@ -13,7 +13,7 @@ import java.util.Set;import java.util.concurrent.locks.ReentrantLock;/**- * Opt-in embedding shortlist for a choice question with many labels. A port of+ * Opt-in embedding shortlist and tournament for a choice question with many labels. A port of* {@code laya.shortlist}.** <p>Every option of a choice shares one {@code head_max_len} budget, so a large label set leaves@@ -42,6 +42,9 @@ import java.util.concurrent.locks.ReentrantLock;* paying per embedding call cares about that last part.* </ul>*+ * <p>{@link #predictTournament} needs no embedder: the decision model answers the labels in groups+ * small enough to keep their tokens, and the group winners meet in one more prediction.+ ** <p>{@code embed_fn_from_agent} is deliberately not ported. It mean-pools the checkpoint's own* encoder, which needs the encoder's hidden states; the ONNX graph this runtime loads exposes* logits, not hidden states. A dedicated bi-encoder passed as an {@link Embedder} shortlists@@ -52,6 +55,9 @@ public final class Shortlist {/** The reference's default: keep twenty labels. */public static final int DEFAULT_K = 20;+ /** The reference's default tournament group: up to 256 labels take one round. */+ public static final int DEFAULT_TOURNAMENT_GROUP = 16;+private Shortlist() {}@@ -236,6 +242,178 @@ public final class Shortlist {return predict(predictor, state, questions, embedder, DEFAULT_K);}+ // ------------------------------------------------------------------ the tournament++ /**+ * What a tournament left of one choice question.+ *+ * @param labels the finalists, in criteria order; every label when no round was played+ * @param total how many labels the question had+ * @param rounds how many elimination rounds the question played+ */+ public record Bracket(List<String> labels, int total, int rounds) {++ public Bracket {+ labels = List.copyOf(labels);+ if (labels.isEmpty()) {+ throw new IllegalArgumentException("a bracket keeps at least one label");+ }+ if (total < labels.size()) {+ throw new IllegalArgumentException(String.format(+ "total is %d but %d labels were kept", total, labels.size()));+ }+ if (rounds < 0) {+ throw new IllegalArgumentException("rounds must not be negative, got " + rounds);+ }+ if (rounds == 0 && labels.size() != total) {+ throw new IllegalArgumentException(String.format(+ "a choice that played no round keeps all %d labels, not %d",+ total, labels.size()));+ }+ }+ }++ /** The final prediction of a tournament, and each choice question's bracket. */+ public record Tournament(Prediction prediction, Map<String, Bracket> tournament) {++ public Tournament {+ if (prediction == null) {+ throw new IllegalArgumentException("prediction must not be null");+ }+ if (tournament == null) {+ throw new IllegalArgumentException("tournament must not be null");+ }+ tournament = Collections.unmodifiableMap(new LinkedHashMap<>(tournament));+ }+ }++ /**+ * Narrow each large choice question by elimination, then predict once more.+ *+ * <p>A choice with more than {@code groupSize} labels is cut, in criteria order, into groups of+ * near-equal size and at most {@code groupSize}. One prediction answers every group of every+ * such question -- the groups go in as separate questions, keyed {@code "0"}, {@code "1"}, ...+ * -- and each group's answer goes through to the next round. Rounds repeat until no choice has+ * more than {@code groupSize} labels left.+ *+ * <p>The final prediction answers every question of the request, with each choice that played+ * a round cut to its finalists; everything else goes to it unchanged, so when nothing needs a+ * round it is the only call. Answers, probabilities and usage are the final call's, so a+ * tournament choice's probabilities are over its finalists only. The caller's map is not+ * modified.+ *+ * @throws IllegalArgumentException if {@code groupSize} is below two+ * @throws IllegalStateException if a round's prediction does not answer a group with one of+ * its labels; the reference fails on such a label at the next+ * cut instead+ */+ public static Tournament predictTournament(Predictor predictor, Object state,+ Map<String, Question> questions, int groupSize) {+ if (predictor == null) {+ throw new IllegalArgumentException("predictor must not be null");+ }+ if (questions == null) {+ throw new IllegalArgumentException("questions must not be null");+ }+ if (groupSize < 2) {+ throw new IllegalArgumentException(+ "group_size must be an integer of at least 2, got " + groupSize);+ }+ Map<String, List<String>> finalists = new LinkedHashMap<>();+ Map<String, Integer> rounds = new LinkedHashMap<>();+ for (Map.Entry<String, Question> entry : questions.entrySet()) {+ Question question = entry.getValue();+ if (question != null && question.type() == Question.Type.CHOICE) {+ finalists.put(entry.getKey(), question.labels());+ rounds.put(entry.getKey(), 0);+ }+ }+ while (true) {+ List<String> owners = new ArrayList<>();+ List<List<String>> groups = new ArrayList<>();+ for (Map.Entry<String, List<String>> entry : finalists.entrySet()) {+ List<List<String>> parts = groups(entry.getValue(), groupSize);+ if (parts.size() > 1) {+ for (List<String> part : parts) {+ owners.add(entry.getKey());+ groups.add(part);+ }+ }+ }+ if (groups.isEmpty()) {+ break;+ }+ Map<String, Question> round = new LinkedHashMap<>();+ for (int i = 0; i < groups.size(); i++) {+ round.put(Integer.toString(i), cut(questions.get(owners.get(i)), groups.get(i)));+ }+ Prediction answered = predictor.predict(state, round);+ Map<String, List<String>> winners = new LinkedHashMap<>();+ for (int i = 0; i < groups.size(); i++) {+ String id = Integer.toString(i);+ Answer answer = answered == null ? null : answered.answer(id);+ if (!(answer instanceof Answer.Choice choice)+ || !groups.get(i).contains(choice.choice())) {+ throw new IllegalStateException(String.format(+ "round question %s of %s was not answered with one of its labels: %s",+ id, PythonJson.repr(owners.get(i)), answer));+ }+ winners.computeIfAbsent(owners.get(i), key -> new ArrayList<>())+ .add(choice.choice());+ }+ for (Map.Entry<String, List<String>> entry : winners.entrySet()) {+ finalists.put(entry.getKey(), entry.getValue());+ rounds.merge(entry.getKey(), 1, Integer::sum);+ }+ }+ Map<String, Question> last = new LinkedHashMap<>(questions);+ Map<String, Bracket> meta = new LinkedHashMap<>();+ for (Map.Entry<String, List<String>> entry : finalists.entrySet()) {+ String id = entry.getKey();+ int played = rounds.get(id);+ if (played > 0) {+ last.put(id, cut(questions.get(id), entry.getValue()));+ }+ meta.put(id, new Bracket(entry.getValue(), questions.get(id).labels().size(), played));+ }+ return new Tournament(predictor.predict(state, last), meta);+ }++ /** A tournament with the reference's default group size. */+ public static Tournament predictTournament(Predictor predictor, Object state,+ Map<String, Question> questions) {+ return predictTournament(predictor, state, questions, DEFAULT_TOURNAMENT_GROUP);+ }++ /**+ * {@code labels} in criteria order, cut into {@code ceil(n / groupSize)} contiguous groups.+ *+ * <p>Group {@code i} is {@code labels[i*n/parts : (i+1)*n/parts]}, the reference's integer+ * split, so sizes differ by at most one and the larger groups fall where it puts them.+ */+ static List<List<String>> groups(List<String> labels, int groupSize) {+ int n = labels.size();+ int parts = (n + groupSize - 1) / groupSize;+ List<List<String>> out = new ArrayList<>(parts);+ for (int i = 0; i < parts; i++) {+ // long, so i * n cannot overflow on a large label set.+ int from = (int) ((long) i * n / parts);+ int to = (int) ((long) (i + 1) * n / parts);+ out.add(List.copyOf(labels.subList(from, to)));+ }+ return out;+ }++ /** The question asked again over these of its labels, in the order given. */+ private static Question cut(Question question, List<String> labels) {+ Map<String, Object> criteria = criteriaOf(question);+ Map<String, Object> subset = new LinkedHashMap<>();+ for (String label : labels) {+ subset.put(label, criteria.get(label));+ }+ return Question.choice(question.instructions(), subset);+ }+// ------------------------------------------------------------------ the ranking/** The one implementation. {@code instructions} may be null, which embeds the state alone. */diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentParityTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentParityTest.javanew file mode 100644index 00000000..1ad716d3--- /dev/null+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentParityTest.java@@ -0,0 +1,150 @@+package com.convaiinnovations.laya;++import static com.convaiinnovations.laya.PredictionGolden.assertAnswers;+import static com.convaiinnovations.laya.PredictionGolden.assertUsage;+import static com.convaiinnovations.laya.PredictionGolden.questionsOf;+import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertSame;+import static org.junit.jupiter.api.Assertions.assertTrue;++import java.io.IOException;+import java.nio.file.Files;+import java.nio.file.Path;+import java.nio.file.Paths;+import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import org.junit.jupiter.api.AfterAll;+import org.junit.jupiter.api.Assumptions;+import org.junit.jupiter.api.DisplayName;+import org.junit.jupiter.api.DynamicTest;+import org.junit.jupiter.api.TestFactory;++/**+ * {@link Shortlist#predictTournament} on the real graph against {@code tournament_golden.json}.+ *+ * <p>Every call is compared: the groups asked (exactly, since the generator refuses a near-tie in+ * any round), each call's answers within {@link PredictionGolden}'s tolerance, and the usage. Skips+ * when the golden was recorded without a graph or the checkpoint and graph are not present.+ */+final class TournamentParityTest {++ private static Agent shared;++ @AfterAll+ static void closeAgent() {+ if (shared != null) {+ shared.close();+ shared = null;+ }+ }++ @SuppressWarnings("unchecked")+ private static Map<String, Object> map(Object value) {+ return (Map<String, Object>) value;+ }++ @SuppressWarnings("unchecked")+ private static List<Object> list(Object value) {+ return (List<Object>) value;+ }++ @TestFactory+ @DisplayName("every recorded tournament replays call for call on the real graph")+ List<DynamicTest> replays() {+ Map<String, Object> golden = Fixtures.load("tournament_golden.json");+ Assumptions.assumeFalse(golden.containsKey("skipped"),+ "tournament_golden.json was recorded without a graph: set " + Fixtures.GRAPH_ENV+ + " and re-run scripts/gen_fixtures.py");+ List<DynamicTest> tests = new ArrayList<>();+ for (Object entry : list(golden.get("cases"))) {+ Map<String, Object> row = map(entry);+ String name = (String) row.get("name");+ tests.add(DynamicTest.dynamicTest(name, () -> {+ Agent agent = agent(golden);+ List<Map<String, Question>> asked = new ArrayList<>();+ List<Prediction> returned = new ArrayList<>();+ Predictor recording = (state, questions) -> {+ asked.add(new LinkedHashMap<>(questions));+ Prediction out = agent.predict(state, questions);+ returned.add(out);+ return out;+ };+ Object groupSize = row.get("group_size");+ Map<String, Question> questions = questionsOf(choicesAsMaps(map(row.get("questions"))));+ Shortlist.Tournament result = groupSize == null+ ? Shortlist.predictTournament(recording, row.get("state"), questions)+ : Shortlist.predictTournament(recording, row.get("state"), questions,+ ((Number) groupSize).intValue());++ List<Object> calls = list(row.get("calls"));+ assertEquals(calls.size(), asked.size(), name + ": calls");+ for (int c = 0; c < calls.size(); c++) {+ Map<String, Object> call = map(calls.get(c));+ Map<String, Object> expected = map(call.get("asked"));+ assertEquals(new ArrayList<>(expected.keySet()),+ new ArrayList<>(asked.get(c).keySet()), name + " call " + c + ": ids");+ for (Map.Entry<String, Object> q : expected.entrySet()) {+ Object labels = map(q.getValue()).get("labels");+ if (labels != null) {+ assertEquals(labels, asked.get(c).get(q.getKey()).labels(),+ name + " call " + c + "." + q.getKey() + ": labels");+ }+ }+ assertAnswers(map(call.get("answers")), returned.get(c).answers());+ assertUsage(map(call.get("usage")), returned.get(c).usage());+ }+ assertSame(returned.get(returned.size() - 1), result.prediction(),+ name + ": answers and usage are the final call's");+ Map<String, Object> meta = map(row.get("tournament"));+ assertEquals(new ArrayList<>(meta.keySet()),+ new ArrayList<>(result.tournament().keySet()));+ for (Map.Entry<String, Object> bracket : meta.entrySet()) {+ Map<String, Object> want = map(bracket.getValue());+ Shortlist.Bracket got = result.tournament().get(bracket.getKey());+ assertEquals(want.get("labels"), got.labels(), name + ": finalists");+ assertEquals(((Number) want.get("n")).intValue(), got.total());+ assertEquals(((Number) want.get("rounds")).intValue(), got.rounds());+ }+ }));+ }+ assertTrue(tests.size() >= 3, "only " + tests.size() + " recorded tournaments");+ return tests;+ }++ /** List criteria on a choice become a map of null descriptions, as the reference reads them. */+ private static Map<String, Object> choicesAsMaps(Map<String, Object> defs) {+ Map<String, Object> out = new LinkedHashMap<>();+ for (Map.Entry<String, Object> entry : defs.entrySet()) {+ Map<String, Object> q = new LinkedHashMap<>(map(entry.getValue()));+ if ("choice".equals(q.get("type")) && q.get("criteria") instanceof List) {+ Map<String, Object> options = new LinkedHashMap<>();+ for (Object label : list(q.get("criteria"))) {+ options.put((String) label, null);+ }+ q.put("criteria", options);+ }+ out.put(entry.getKey(), q);+ }+ return out;+ }++ private static synchronized Agent agent(Map<String, Object> golden) throws IOException {+ if (shared == null) {+ String checkpoint = (String) golden.getOrDefault("checkpoint", "multilingual");+ Path model = Fixtures.checkpoint(checkpoint);+ Assumptions.assumeTrue(model != null, Fixtures.missingCheckpoint(checkpoint));+ String graph = System.getenv(Fixtures.GRAPH_ENV);+ Assumptions.assumeTrue(graph != null && !graph.isBlank(),+ "set " + Fixtures.GRAPH_ENV + " to the exported laya.onnx this golden was"+ + " recorded from");+ Path graphPath = Paths.get(graph);+ Path directory = Files.isDirectory(graphPath) ? graphPath : graphPath.getParent();+ Assumptions.assumeTrue(directory != null && Files.isDirectory(directory),+ Fixtures.GRAPH_ENV + " does not name a graph: " + graph);+ shared = Agent.open(model, directory);+ }+ return shared;+ }+}diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentTest.javanew file mode 100644index 00000000..e6bb0dcf--- /dev/null+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentTest.java@@ -0,0 +1,384 @@+package com.convaiinnovations.laya;++import static org.junit.jupiter.api.Assertions.assertEquals;+import static org.junit.jupiter.api.Assertions.assertSame;+import static org.junit.jupiter.api.Assertions.assertThrows;+import static org.junit.jupiter.api.Assertions.assertTrue;++import java.nio.charset.StandardCharsets;+import java.util.ArrayList;+import java.util.LinkedHashMap;+import java.util.List;+import java.util.Map;+import org.junit.jupiter.api.DisplayName;+import org.junit.jupiter.api.DynamicTest;+import org.junit.jupiter.api.Test;+import org.junit.jupiter.api.TestFactory;++/**+ * {@link Shortlist#predictTournament} against the calls recorded from+ * {@code laya.shortlist.predict_tournament}.+ *+ * <p>Both sides answer with the same stand-in model: every choice goes to the label with the+ * lowest unsigned FNV-1a of {@code rank:<label>}. That makes every round reproducible, so the+ * fixture pins the groups asked, their order, the winners carried forward and the final cut.+ */+class TournamentTest {++ private static final long FNV_OFFSET = 0xCBF29CE484222325L;+ private static final long FNV_PRIME = 0x100000001B3L;++ private static Map<String, Object> fixture() {+ return Fixtures.load("tournament.json");+ }++ @SuppressWarnings("unchecked")+ private static Map<String, Object> map(Object value) {+ return (Map<String, Object>) value;+ }++ @SuppressWarnings("unchecked")+ private static List<Object> list(Object value) {+ return (List<Object>) value;+ }++ private static List<String> strings(Object value) {+ List<String> out = new ArrayList<>();+ for (Object item : list(value)) {+ out.add((String) item);+ }+ return out;+ }++ private static long rank(String label) {+ long hash = FNV_OFFSET;+ for (byte value : ("rank:" + label).getBytes(StandardCharsets.UTF_8)) {+ hash = (hash ^ (value & 0xFFL)) * FNV_PRIME;+ }+ return hash;+ }++ private static Usage noUsage() {+ return new Usage(0, 0, 0, 0, false, List.of(), Map.of());+ }++ /** The stand-in model, recording each call: choices only, like the generator's. */+ private static final class Ranker implements Predictor {++ final List<Map<String, Question>> calls = new ArrayList<>();+ final List<Prediction> returned = new ArrayList<>();+ final List<Object> states = new ArrayList<>();++ @Override+ public Prediction predict(Object state, Map<String, Question> questions) {+ calls.add(new LinkedHashMap<>(questions));+ states.add(state);+ Map<String, Answer> answers = new LinkedHashMap<>();+ for (Map.Entry<String, Question> entry : questions.entrySet()) {+ if (entry.getValue().type() != Question.Type.CHOICE) {+ continue;+ }+ String best = null;+ for (String label : entry.getValue().labels()) {+ if (best == null || Long.compareUnsigned(rank(label), rank(best)) < 0) {+ best = label;+ }+ }+ answers.put(entry.getKey(), new Answer.Choice(best, Map.of(best, 1.0), 1, 1, 0));+ }+ Prediction out = new Prediction("stub-" + calls.size(), answers, noUsage());+ returned.add(out);+ return out;+ }+ }++ /** A recorded question; list criteria become a map of null descriptions, as the reference does. */+ private static Question question(Map<String, Object> spec) {+ String type = (String) spec.get("type");+ String instructions = (String) spec.get("instructions");+ Object criteria = spec.get("criteria");+ if ("choice".equals(type)) {+ Map<String, Object> options = new LinkedHashMap<>();+ if (criteria instanceof List) {+ for (Object label : list(criteria)) {+ options.put((String) label, null);+ }+ } else {+ options.putAll(map(criteria));+ }+ return Question.choice(instructions, options);+ }+ if ("score".equals(type)) {+ return Question.score(instructions, list(criteria));+ }+ return Question.noul(instructions);+ }++ private static Map<String, Question> questions(Map<String, Object> specs) {+ Map<String, Question> out = new LinkedHashMap<>();+ for (Map.Entry<String, Object> entry : specs.entrySet()) {+ out.put(entry.getKey(), question(map(entry.getValue())));+ }+ return out;+ }++ private static Map<String, Question> oneChoice(int n) {+ Map<String, Object> criteria = new LinkedHashMap<>();+ for (int i = 0; i < n; i++) {+ criteria.put(String.format("l%04d", i), null);+ }+ return Map.of("q", Question.choice("pick", criteria));+ }++ // ------------------------------------------------------------------ golden parity++ @Test+ @DisplayName("the stand-in model ranks labels as the generator did")+ void rankMatchesTheGenerator() {+ Map<String, Object> probe = map(fixture().get("rank_probe"));+ assertTrue(probe.size() >= 3, "only " + probe.size() + " probe labels");+ for (Map.Entry<String, Object> entry : probe.entrySet()) {+ assertEquals(entry.getValue(), Long.toUnsignedString(rank(entry.getKey())),+ "rank of " + entry.getKey());+ }+ assertEquals(((Number) fixture().get("default_group_size")).intValue(),+ Shortlist.DEFAULT_TOURNAMENT_GROUP);+ }++ @TestFactory+ @DisplayName("every recorded split: group sizes per round, finalists and the final choice")+ List<DynamicTest> splits() {+ List<DynamicTest> tests = new ArrayList<>();+ for (Object entry : list(fixture().get("splits"))) {+ Map<String, Object> row = map(entry);+ int groupSize = ((Number) row.get("group_size")).intValue();+ int n = ((Number) row.get("n")).intValue();+ tests.add(DynamicTest.dynamicTest("n=" + n + " group_size=" + groupSize, () -> {+ Ranker ranker = new Ranker();+ Shortlist.Tournament result = Shortlist.predictTournament(ranker, "s",+ oneChoice(n), groupSize);+ List<Object> rounds = list(row.get("rounds"));+ assertEquals(rounds.size() + 1, ranker.calls.size(), "calls");+ for (int r = 0; r < rounds.size(); r++) {+ List<Integer> sizes = new ArrayList<>();+ for (Question asked : ranker.calls.get(r).values()) {+ sizes.add(asked.labels().size());+ }+ List<Integer> expected = new ArrayList<>();+ for (Object size : list(rounds.get(r))) {+ expected.add(((Number) size).intValue());+ }+ assertEquals(expected, sizes, "group sizes of round " + r);+ }+ Shortlist.Bracket bracket = result.tournament().get("q");+ assertEquals(strings(row.get("finalists")), bracket.labels(), "finalists");+ assertEquals(rounds.size(), bracket.rounds(), "rounds");+ assertEquals(n, bracket.total(), "total");+ assertEquals(row.get("choice"),+ ((Answer.Choice) result.prediction().answer("q")).choice(), "choice");+ }));+ }+ assertTrue(tests.size() >= 100, "only " + tests.size() + " splits recorded");+ return tests;+ }++ @TestFactory+ @DisplayName("every recorded tournament asks the same calls and reports the same result")+ List<DynamicTest> cases() {+ List<DynamicTest> tests = new ArrayList<>();+ String state = (String) fixture().get("state");+ for (Object entry : list(fixture().get("cases"))) {+ Map<String, Object> row = map(entry);+ String name = (String) row.get("name");+ tests.add(DynamicTest.dynamicTest(name, () -> {+ Map<String, Question> questions = questions(map(row.get("questions")));+ Map<String, Question> original = new LinkedHashMap<>(questions);+ Ranker ranker = new Ranker();+ Object groupSize = row.get("group_size");+ Shortlist.Tournament result = groupSize == null+ ? Shortlist.predictTournament(ranker, state, questions)+ : Shortlist.predictTournament(ranker, state, questions,+ ((Number) groupSize).intValue());++ List<Object> calls = list(row.get("calls"));+ assertEquals(calls.size(), ranker.calls.size(), name + ": calls");+ for (int c = 0; c < calls.size(); c++) {+ assertAsked(name + " call " + c, map(calls.get(c)), ranker.calls.get(c));+ assertSame(state, ranker.states.get(c), name + ": the state is passed through");+ }+ assertEquals(row.get("model"), result.prediction().model(),+ name + ": the result is the final call's");+ assertSame(ranker.returned.get(ranker.returned.size() - 1), result.prediction());+ Map<String, Object> answers = map(row.get("answers"));+ assertEquals(new ArrayList<>(answers.keySet()),+ new ArrayList<>(result.prediction().answers().keySet()));+ for (Map.Entry<String, Object> answer : answers.entrySet()) {+ assertEquals(answer.getValue(), ((Answer.Choice) result.prediction()+ .answer(answer.getKey())).choice(), name + "." + answer.getKey());+ }+ Map<String, Object> meta = map(row.get("tournament"));+ assertEquals(new ArrayList<>(meta.keySet()),+ new ArrayList<>(result.tournament().keySet()), name + ": brackets");+ for (Map.Entry<String, Object> expected : meta.entrySet()) {+ Map<String, Object> want = map(expected.getValue());+ Shortlist.Bracket got = result.tournament().get(expected.getKey());+ String at = name + "." + expected.getKey();+ assertEquals(strings(want.get("labels")), got.labels(), at + ".labels");+ assertEquals(((Number) want.get("n")).intValue(), got.total(), at + ".n");+ assertEquals(((Number) want.get("rounds")).intValue(), got.rounds(),+ at + ".rounds");+ }+ assertEquals(Boolean.TRUE, row.get("caller_questions_unmutated"));+ assertEquals(original, questions, name + ": the caller's map was modified");+ }));+ }+ // Floors: a multi-round case and a case where questions play different numbers of rounds.+ int multiRound = 0;+ int mixedRounds = 0;+ for (Object entry : list(fixture().get("cases"))) {+ List<Integer> played = new ArrayList<>();+ for (Object bracket : map(map(entry).get("tournament")).values()) {+ played.add(((Number) map(bracket).get("rounds")).intValue());+ }+ multiRound += played.stream().anyMatch(r -> r > 1) ? 1 : 0;+ mixedRounds += played.stream().distinct().count() > 1 ? 1 : 0;+ }+ assertTrue(multiRound >= 1, "no case plays more than one round");+ assertTrue(mixedRounds >= 1, "no case mixes round counts");+ return tests;+ }++ private static void assertAsked(String at, Map<String, Object> expected,+ Map<String, Question> seen) {+ assertEquals(new ArrayList<>(expected.keySet()), new ArrayList<>(seen.keySet()),+ at + ": question ids and order");+ for (Map.Entry<String, Object> entry : expected.entrySet()) {+ Map<String, Object> want = map(entry.getValue());+ Question got = seen.get(entry.getKey());+ String where = at + "." + entry.getKey();+ assertEquals(want.get("type"), got.type().wireName(), where + ".type");+ assertEquals(want.get("instructions"), got.instructions(), where + ".instructions");+ if (want.containsKey("labels")) {+ assertEquals(strings(want.get("labels")), got.labels(), where + ".labels");+ }+ if (want.containsKey("criteria")) {+ assertEquals(map(want.get("criteria")), got.spec().get("criteria"),+ where + ".criteria");+ }+ }+ }++ @TestFactory+ @DisplayName("a group size below two is refused with the reference's message")+ List<DynamicTest> refusals() {+ List<DynamicTest> tests = new ArrayList<>();+ for (Object entry : list(fixture().get("refusals"))) {+ Map<String, Object> row = map(entry);+ int bad = ((Number) row.get("group_size")).intValue();+ tests.add(DynamicTest.dynamicTest("group_size=" + bad, () -> {+ assertEquals("ValueError", row.get("error"));+ IllegalArgumentException refused = assertThrows(IllegalArgumentException.class,+ () -> Shortlist.predictTournament(new Ranker(), "s", Map.of(), bad));+ assertEquals(row.get("message"), refused.getMessage());+ }));+ }+ assertTrue(tests.size() >= 3);+ return tests;+ }++ // ------------------------------------------------------------------ model-free behaviour++ @Test+ @DisplayName("77 labels at the default group go out as five balanced groups in criteria order")+ void balancedGroups() {+ List<String> labels = new ArrayList<>(oneChoice(77).get("q").labels());+ List<List<String>> groups = Shortlist.groups(labels, 16);+ List<Integer> sizes = new ArrayList<>();+ List<String> joined = new ArrayList<>();+ for (List<String> group : groups) {+ sizes.add(group.size());+ joined.addAll(group);+ }+ assertEquals(List.of(15, 15, 16, 15, 16), sizes);+ assertEquals(labels, joined, "groups are contiguous slices in criteria order");+ assertEquals(1, Shortlist.groups(labels.subList(0, 16), 16).size());+ }++ @Test+ @DisplayName("a consistent ranker's best label wins from any position")+ void bestLabelWinsFromAnyPosition() {+ for (int[] shape : new int[][] {{77, 2}, {257, 3}}) {+ int n = shape[0];+ List<String> labels = new ArrayList<>(oneChoice(n).get("q").labels());+ for (String best : List.of(labels.get(0), labels.get(n / 2), labels.get(n - 1))) {+ List<Map<String, Question>> calls = new ArrayList<>();+ Predictor favours = (s, qs) -> {+ calls.add(qs);+ Map<String, Answer> answers = new LinkedHashMap<>();+ for (Map.Entry<String, Question> q : qs.entrySet()) {+ List<String> options = q.getValue().labels();+ String pick = options.contains(best) ? best : options.get(0);+ answers.put(q.getKey(), new Answer.Choice(pick, Map.of(), 1, 1, 0));+ }+ return new Prediction("stub", answers, noUsage());+ };+ Shortlist.Tournament out = Shortlist.predictTournament(favours, "s", oneChoice(n));+ assertEquals(best, ((Answer.Choice) out.prediction().answer("q")).choice(),+ n + " labels, best " + best);+ assertEquals(shape[1], calls.size(), n + " labels, best " + best + ": calls");+ }+ }+ }++ @Test+ @DisplayName("nothing to narrow: one call, with the caller's questions as given")+ void nothingToNarrowIsOneCall() {+ Map<String, Question> asked = new LinkedHashMap<>();+ asked.put("small", Question.choice("pick", Map.of("x", "")));+ asked.put("level", Question.score("rate", List.of("low", "high")));+ Ranker ranker = new Ranker();+ Shortlist.Tournament out = Shortlist.predictTournament(ranker, "s", asked, 2);+ assertEquals(1, ranker.calls.size());+ assertEquals(asked, ranker.calls.get(0));+ assertSame(asked.get("small"), ranker.calls.get(0).get("small"));+ assertEquals(0, out.tournament().get("small").rounds());+ assertEquals(List.of("small"), new ArrayList<>(out.tournament().keySet()));+ }++ @Test+ @DisplayName("a round answer outside its group, or missing, is refused")+ void badRoundAnswerIsRefused() {+ Predictor outside = (s, qs) -> new Prediction("stub", Map.of("0",+ new Answer.Choice("l0039", Map.of(), 1, 1, 0)), noUsage());+ IllegalStateException wrong = assertThrows(IllegalStateException.class,+ () -> Shortlist.predictTournament(outside, "s", oneChoice(40), 16));+ assertTrue(wrong.getMessage().contains("round question 0 of 'q'"), wrong.getMessage());+ Predictor silent = (s, qs) -> new Prediction("stub", Map.of(), noUsage());+ assertThrows(IllegalStateException.class,+ () -> Shortlist.predictTournament(silent, "s", oneChoice(40), 16));+ Predictor nothing = (s, qs) -> null;+ assertThrows(IllegalStateException.class,+ () -> Shortlist.predictTournament(nothing, "s", oneChoice(40), 16));+ assertThrows(IllegalArgumentException.class,+ () -> Shortlist.predictTournament(nothing, "s", oneChoice(3), 16),+ "a null final prediction");+ }++ @Test+ @DisplayName("null arguments and contradictory brackets are refused")+ void refusesNullsAndBadBrackets() {+ assertThrows(IllegalArgumentException.class,+ () -> Shortlist.predictTournament(null, "s", Map.of()));+ assertThrows(IllegalArgumentException.class,+ () -> Shortlist.predictTournament(new Ranker(), "s", null));+ assertThrows(IllegalArgumentException.class,+ () -> new Shortlist.Bracket(List.of(), 0, 0));+ assertThrows(IllegalArgumentException.class,+ () -> new Shortlist.Bracket(List.of("a", "b"), 1, 1));+ assertThrows(IllegalArgumentException.class,+ () -> new Shortlist.Bracket(List.of("a"), 2, 0));+ assertThrows(IllegalArgumentException.class,+ () -> new Shortlist.Bracket(List.of("a"), 2, -1));+ assertEquals(1, new Shortlist.Bracket(List.of("a"), 2, 1).rounds());+ }+}diff --git a/laya-java/scripts/gen_fixtures.py b/laya-java/scripts/gen_fixtures.pyindex 0a359c2e..429bc4fa 100644--- a/laya-java/scripts/gen_fixtures.py+++ b/laya-java/scripts/gen_fixtures.py@@ -1104,6 +1104,233 @@ def shortlist():return payload+def _tournament_rank(label):+ """The stand-in model's preference: the lowest FNV-1a of `rank:<label>` wins any group."""+ return _fnv1a(("rank:%s" % label).encode("utf-8"))+++class _TournamentRecorder:+ """Answers every choice with its best-ranked label and records each call, in order."""++ def __init__(self):+ self.calls = []++ def predict(self, state, questions, **kwargs):+ from laya import shortlist as mod++ self.calls.append({qid: dict(qdef) for qid, qdef in questions.items()})+ answers = {}+ for qid, qdef in questions.items():+ if qdef.get("type") == "choice":+ labels = [key for key, _value in mod._criteria_items(qdef["criteria"])]+ answers[qid] = {"type": "choice", "choice": min(labels, key=_tournament_rank)}+ return {"model": "stub-%d" % len(self.calls), "answers": answers}+++def _asked(questions):+ """A recorded call: each question's type, instructions, labels, and dict criteria verbatim."""+ from laya import shortlist as mod++ out = {}+ for qid, qdef in questions.items():+ entry = {"type": qdef.get("type"), "instructions": qdef.get("instructions")}+ if qdef.get("type") == "choice":+ entry["labels"] = [key for key, _value in mod._criteria_items(qdef["criteria"])]+ if isinstance(qdef["criteria"], dict):+ entry["criteria"] = qdef["criteria"]+ out[qid] = entry+ return out+++def tournament():+ """`predict_tournament`'s grouping, rounds and final call, against a deterministic stub model.++ The stub prefers the label with the lowest FNV-1a of `rank:<label>`, reproducible in any+ language, so every call the reference makes -- the groups of each round, their order, the+ winners carried forward and the final cut -- can be replayed by a port and compared exactly.+ """+ from laya import shortlist as mod++ state = "My card was charged twice for the same order last Tuesday."++ def run(questions, group_size=None):+ recorder = _TournamentRecorder()+ before = json.dumps(questions, sort_keys=False)+ kwargs = {} if group_size is None else {"group_size": group_size}+ result = mod.predict_tournament(recorder, state, questions, **kwargs)+ return recorder, result, before == json.dumps(questions, sort_keys=False)++ # Every split size, from the calls the reference actually makes on one bare-label choice.+ splits = []+ sweep = [(g, n) for g in (2, 3, 5) for n in range(1, 41)]+ sweep += [(16, n) for n in (1, 15, 16, 17, 31, 32, 33, 77, 150, 255, 256, 257, 300, 1000)]+ for group_size, n in sweep:+ labels = ["l%04d" % i for i in range(n)]+ recorder, result, _same = run(+ {"q": {"type": "choice", "instructions": "pick", "criteria": labels}}, group_size)+ splits.append({+ "group_size": group_size,+ "n": n,+ "rounds": [[len(q["criteria"]) for q in call.values()] for call in recorder.calls[:-1]],+ "finalists": result["tournament"]["q"]["labels"],+ "choice": result["answers"]["q"]["choice"],+ })++ big = {label: "about " + label for label in ["l%02d" % i for i in range(20)]}+ banking = [+ "card_arrival", "card_delivery_estimate", "card_not_working", "cash_withdrawal_charge",+ "declined_card_payment", "direct_debit_payment_not_recognised", "exchange_rate",+ "failed_transfer", "lost_or_stolen_card", "pending_card_payment",+ "refund_not_showing_up", "request_refund", "reverted_card_payment", "top_up_failed",+ "transfer_fee_charged", "verify_my_identity", "wrong_amount_of_cash_received",+ ]+ cases = []+ for name, questions, group_size in [+ ("default-group-77", {"q": {"type": "choice", "instructions": "pick",+ "criteria": ["l%03d" % i for i in range(77)]}}, None),+ ("mixed-group-8", {+ "big": {"type": "choice", "instructions": "pick", "criteria": big},+ "small": {"type": "choice", "instructions": "pick", "criteria": ["x", "y"]},+ "level": {"type": "score", "instructions": "rate", "criteria": ["low", "high"]},+ "urgent": {"type": "noul", "instructions": "Is this urgent?"},+ }, 8),+ ("nothing-to-narrow", {+ "small": {"type": "choice", "instructions": "pick", "criteria": ["x", "y"]},+ "level": {"type": "score", "instructions": "rate", "criteria": ["low", "high"]},+ }, None),+ ("two-choices-different-rounds", {+ "wide": {"type": "choice", "instructions": "Which one?",+ "criteria": ["w%03d" % i for i in range(300)]},+ "narrow": {"type": "choice", "instructions": "Which other?",+ "criteria": ["n%02d" % i for i in range(40)]},+ }, None),+ ("described-criteria", {"intent": {+ "type": "choice", "instructions": "What does the customer want?",+ "criteria": dict([(label, None if i % 4 == 0 else ("" if i % 4 == 1 else+ (i if i % 4 == 2 else label.replace("_", " "))))+ for i, label in enumerate(banking)])}}, 3),+ ("group-of-one", {"q": {"type": "choice", "instructions": "pick",+ "criteria": ["e", "d", "c", "b", "a"]}}, 2),+ ("criteria-order-not-sorted", {"q": {"type": "choice", "instructions": "pick",+ "criteria": list(reversed(banking))}}, 4),+ ]:+ recorder, result, unchanged = run(questions, group_size)+ cases.append({+ "name": name,+ "group_size": group_size,+ "questions": questions,+ "calls": [_asked(call) for call in recorder.calls],+ "model": result["model"],+ "answers": {qid: answer["choice"] for qid, answer in result["answers"].items()},+ "tournament": result["tournament"],+ "caller_questions_unmutated": unchanged,+ })++ refusals = []+ for bad in (1, 0, -3):+ try:+ mod.predict_tournament(_TournamentRecorder(), state, {}, group_size=bad)+ refusals.append({"group_size": bad, "error": None})+ except Exception as failure:+ refusals.append({"group_size": bad, "error": type(failure).__name__,+ "message": str(failure)})++ return {+ "default_group_size": mod.DEFAULT_TOURNAMENT_GROUP,+ "state": state,+ "rank_probe": {label: str(_tournament_rank(label)) for label in ["", "a", "l0001", "注文"]},+ "splits": splits,+ "cases": cases,+ "refusals": refusals,+ }+++TOURNAMENT_GOLDEN_CASES = [+ ("banking-default-group", "My new card still has not arrived after two weeks.", {+ "intent": {"type": "choice", "instructions": "What does the customer want?",+ "criteria": [+ "activate_my_card", "age_limit", "apple_pay_or_google_pay",+ "atm_support", "automatic_top_up", "balance_not_updated_after_bank_transfer",+ "beneficiary_not_allowed", "cancel_transfer", "card_about_to_expire",+ "card_acceptance", "card_arrival", "card_delivery_estimate",+ "card_linking", "card_not_working", "card_payment_fee_charged",+ "card_payment_not_recognised", "card_swallowed", "cash_withdrawal_charge",+ "change_pin", "compromised_card", "contactless_not_working",+ "country_support", "declined_card_payment", "declined_cash_withdrawal",+ "declined_transfer", "edit_personal_details", "exchange_charge",+ "exchange_rate", "failed_transfer", "fiat_currency_support",+ "get_physical_card", "lost_or_stolen_card", "lost_or_stolen_phone",+ "order_physical_card", "passcode_forgotten", "pending_card_payment",+ "refund_not_showing_up", "request_refund", "terminate_account",+ "top_up_failed"]},+ }, None),+ ("mixed-group-8", "We were billed twice for March and want a refund today.", {+ "dept": {"type": "choice", "instructions": "Which team owns this?",+ "criteria": {"billing": "invoices and charges", "refunds": "money sent back",+ "fraud": "unauthorised activity", "cards": "card problems",+ "transfers": "bank transfers", "loans": "credit and loans",+ "savings": "savings products", "kyc": "identity checks",+ "support": "general help", "legal": "complaints and law",+ "sales": "new products", "tech": "app bugs"}},+ "urgent": {"type": "noul", "instructions": "This needs a human today."},+ }, 8),+ ("nothing-to-narrow", "The app crashes when I open statements.", {+ "intent": {"type": "choice", "instructions": "What is wrong?",+ "criteria": {"bug": "the app fails", "billing": "a charge", "other": "else"}},+ }, None),+]+++def tournament_golden():+ """`predict_tournament` on a real graph: every call's answers and the final usage.++ Needs an exported graph, like `predict.json`, and records `skipped` without one. A round's+ winners decide what the next call is asked, so each case is checked here to have a decisive+ answer in every group: a near-tie would make the replay platform-dependent.+ """+ import os++ graph = os.environ.get("LAYA_ONNX_GRAPH")+ model = os.environ.get("LAYA_PREDICT_MODEL", "multilingual")+ rig = os.environ.get("LAYA_FIXTURE_CHECKPOINTS", DEFAULT_CHECKPOINT_ROOT)+ if not graph or not os.path.exists(graph):+ return {"skipped": "set LAYA_ONNX_GRAPH to an exported laya.onnx to record this family"}+ checkpoint = os.path.join(rig, model)+ for needed in ("rl_agent_config.json", "tokenizer/tokenizer.json"):+ if not os.path.isfile(os.path.join(checkpoint, *needed.split("/"))):+ return {"skipped": "set LAYA_FIXTURE_CHECKPOINTS to a root whose %s checkpoint has "+ "%s, to record this family" % (model, needed)}+ from laya import shortlist as mod+ from laya.onnx_agent import ONNXAgent++ agent = ONNXAgent(checkpoint, onnx_path=graph)++ class Recording:+ def __init__(self):+ self.calls = []++ def predict(self, state, questions, **kwargs):+ result = agent.predict(state, questions, **kwargs)+ self.calls.append({"asked": _asked(questions), "answers": result["answers"],+ "usage": result["usage"]})+ return result++ cases = []+ for name, state, questions, group_size in TOURNAMENT_GOLDEN_CASES:+ recording = Recording()+ kwargs = {} if group_size is None else {"group_size": group_size}+ result = mod.predict_tournament(recording, state, questions, **kwargs)+ for call in recording.calls[:-1]:+ for qid, answer in call["answers"].items():+ top = sorted(answer["probabilities"].values(), reverse=True)+ if len(top) > 1 and top[0] - top[1] <= 1e-3:+ raise AssertionError("%s: round answer %s is not decisive" % (name, qid))+ cases.append({"name": name, "state": state, "group_size": group_size,+ "questions": questions, "calls": recording.calls,+ "model": result["model"], "tournament": result["tournament"]})+ return {"checkpoint": model, "graph": os.path.basename(graph), "cases": cases}++# How many decimals of a cosine are safe to record. The reference computes its scores with# `np.dot(matrix, vector)`, a BLAS matrix-vector product, and how BLAS blocks that accumulation# differs between implementations -- measured here: on Apple Accelerate one row of a four-row@@ -4957,6 +5184,8 @@ FAMILIES = {"presets.json": presets,"router.json": router,"shortlist.json": shortlist,+ "tournament.json": tournament,+ "tournament_golden.json": tournament_golden,"tokenizer_ids.json": tokenizer_ids,"decode_text.json": decode_text,"sequences.json": sequences,