Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(java): port predict_tournament as Shortlist.predictTournament

Adeshina Lasisi71e43b75e5f7
diff --git a/.github/workflows/java.yml b/.github/workflows/java.yml
index 5b65c99f..615161a3 100644
--- a/.github/workflows/java.yml
+++ b/.github/workflows/java.yml
@@ -154,17 +154,17 @@ jobs:
# Without LAYA_CHECKPOINTS the checkpoint-backed factories abort by assumption and the rest
# run. `--min-tests` is what stops an aborted suite from looking like a pass.
#
- # Measured on the merged tree: 980 register here, 963 run, 17 abort on a checkpoint this
+ # Measured on the merged tree: 1133 register here, 1113 run, 20 abort on a checkpoint this
# lane does not have. Exact, not padded -- the floor stood at 680 long enough for the real
# figure to pass it by, which is how a floor stops being one. ONE below the JDK-matrix
# lane's floor on purpose: that lane passes `-PtestJavaVersion`, which is what makes
- # `TestJvmVersionTest` run instead of skip, so it executes 964. Same suite, one more test,
+ # `TestJvmVersionTest` run instead of skip, so it executes 1114. Same suite, one more test,
# because the flag enables it. The step below is where the number is justified.
- name: Tests that need no model
working-directory: laya-java
run: ./gradlew --no-daemon test
- # Measured on the merged tree, Corretto 17, no checkpoints: 980 registered, 17 abort,
- # 963 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
+ # Measured on the merged tree, Corretto 17, no checkpoints: 1133 registered, 20 abort,
+ # 1113 RUN -- the floor counts what ran, so the aborted checkpoint-backed cases are not in
# it. The floor is EXACT, with no margin, which is a change from the margin-of-4 this
# replaced. A margin only binds classes larger than itself, and three classes in this lane
# run exactly ONE test (`DecoderTest`, `TypedDecisionsParityTest`, `WindowPlanTest`), so no
@@ -179,7 +179,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 963 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 1113 --allow-aborted
# The samples are single-file source programs, deliberately NOT a Gradle module: a
# `samples` subproject would inherit the root build's MavenPublication and
@@ -335,7 +335,7 @@ jobs:
# dropped, Gradle exits 0 and the floor alone exits 0 on 488 tests, having run JDK 17 twice
# and reported it as two JDKs. With `--require-passed` it exits 1.
#
- # 964, not the build lane's 963, and the one is the point: this lane runs one test MORE
+ # 1114, not the build lane's 1113, and the one is the point: this lane runs one test MORE
# than that one, because `TestJvmVersionTest` is `@EnabledIfSystemProperty` and only
# enabled here. Both are EXACT, with no margin, for the reason the build lane gives --
# three classes here run exactly one test, so no positive margin can bind them.
@@ -356,7 +356,7 @@ jobs:
if: ${{ !cancelled() }}
run: >
python laya-java/scripts/check_test_results.py
- laya-java/laya-java/build/test-results/test --min-tests 964 --allow-aborted
+ laya-java/laya-java/build/test-results/test --min-tests 1114 --allow-aborted
--require-passed TestJvmVersionTest
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
if: ${{ failure() }}
@@ -459,9 +459,12 @@ jobs:
- name: predict
checkpoints: multilingual
graph-model: multilingual
- families: --only predict.json
- tests: --tests com.convaiinnovations.laya.PredictParityTest
- min-tests: 9
+ families: --only predict.json --only tournament_golden.json
+ tests: >-
+ --tests com.convaiinnovations.laya.PredictParityTest
+ --tests com.convaiinnovations.laya.TournamentParityTest
+ # Measured locally: 9 PredictParityTest + 3 TournamentParityTest. Exact.
+ min-tests: 12
timeout: 75
# Weight-bearing, and a SEPARATE cell rather than a second family in the one above.
# Three reasons, in order of how much they cost:
@@ -573,7 +576,7 @@ jobs:
# Re-recorded from the Python code at THIS commit. `--strict` makes a family this cell could
# not regenerate an error rather than a skip: here a skip means the download or the export
# failed, not that the machine has no model. It is scoped by `--only` to the families this
- # cell owns, and between the two cells all four checkpoint-backed families are covered.
+ # cell owns, and between the cells every checkpoint-backed family is covered.
- name: Regenerate this cell's fixtures from Python
env:
LAYA_FIXTURE_CHECKPOINTS: ${{ github.workspace }}/laya-java/.work/checkpoints
diff --git a/.github/workflows/release-java.yml b/.github/workflows/release-java.yml
index c9e6629f..774a74b1 100644
--- a/.github/workflows/release-java.yml
+++ b/.github/workflows/release-java.yml
@@ -67,7 +67,7 @@ jobs:
# that lane already covered.
- name: Model-free tests
run: ./gradlew --no-daemon test
- # 963 of the 980 registered tests run without a checkpoint. This floor has been raised
+ # 1113 of the 1133 registered tests run without a checkpoint. This floor has been raised
# as the model-free suite grew -- 440 left 191 of slack, 625 left 60 -- because a floor
# that does not bind is not a gate: under 625 the whole confidence-gate family of 26
# tests could have vanished from this lane with the gate still green.
@@ -83,7 +83,7 @@ jobs:
- name: The suite was not vacuous
run: >
python3 scripts/check_test_results.py laya-java/build/test-results/test
- --min-tests 963 --allow-aborted
+ --min-tests 1113 --allow-aborted
- name: Build the jars
run: ./gradlew --no-daemon build -x test
diff --git a/laya-java/CHANGELOG.md b/laya-java/CHANGELOG.md
index b0d6dad1..cc85c539 100644
--- a/laya-java/CHANGELOG.md
+++ b/laya-java/CHANGELOG.md
@@ -110,6 +110,8 @@ see [Installing](README.md#installing).
- `Router`: checkpoint selection with its reason strings, plus a load-and-evict lifecycle with
leases, so a checkpoint in use is never closed under a caller.
- `Shortlist`: cosine ranking over a caller's embedder, with an LRU cache and numpy's tie order.
+- `Shortlist.predictTournament`: elimination rounds over a large choice, a port of
+ `predict_tournament`, with no embedder.
- `LayaEmail`: `cleanEmailBody` and `emailState`, with the English, Portuguese, Spanish and French
marker sets, and `emailQuestions` re-exported from `Presets`.
- `Decisions`: schema-driven decisions — `decide` and `decideBatch` over a JSON schema. An
diff --git a/laya-java/README.md b/laya-java/README.md
index 6787b608..e6f6361b 100644
--- a/laya-java/README.md
+++ b/laya-java/README.md
@@ -247,6 +247,17 @@ passthrough: the labels come back in order and **the embedder is never called**.
Both `Agent` and `Router` implement `Predictor`, so shortlisting works identically against a fixed
checkpoint or a routed one.
+Without an embedder, `Shortlist.predictTournament` lets the model narrow the set itself: labels are
+answered in near-equal groups of at most 16 (one prediction per round covers every group of every
+question), and the winners meet in one final prediction.
+
+```java
+Shortlist.Tournament out = Shortlist.predictTournament(agent, state, questions);
+
+out.prediction(); // the final call's answers and usage, over the finalists
+out.tournament().get("intent"); // the finalists, the label count and the rounds played
+```
+
## Cleaning an email
`laya.email`'s cleaner and state builder. The markers cover English, Portuguese, Spanish and
diff --git a/laya-java/fixtures/tournament.json b/laya-java/fixtures/tournament.json
new file mode 100644
index 00000000..c12a0065
--- /dev/null
+++ b/laya-java/fixtures/tournament.json
@@ -0,0 +1,5525 @@
+{
+ "default_group_size": 16,
+ "state": "My card was charged twice for the same order last Tuesday.",
+ "rank_probe": {
+ "": "5951527874303425639",
+ "a": "676785770819501618",
+ "l0001": "16490875346976899246",
+ "注文": "11652523757631009561"
+ },
+ "splits": [
+ {
+ "group_size": 2,
+ "n": 1,
+ "rounds": [],
+ "finalists": [
+ "l0000"
+ ],
+ "choice": "l0000"
+ },
+ {
+ "group_size": 2,
+ "n": 2,
+ "rounds": [],
+ "finalists": [
+ "l0000",
+ "l0001"
+ ],
+ "choice": "l0001"
+ },
+ {
+ "group_size": 2,
+ "n": 3,
+ "rounds": [
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0000",
+ "l0002"
+ ],
+ "choice": "l0002"
+ },
+ {
+ "group_size": 2,
+ "n": 4,
+ "rounds": [
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0001",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 2,
+ "n": 5,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0000",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 2,
+ "n": 6,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0001",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 2,
+ "n": 7,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0002",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 2,
+ "n": 8,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0007"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 2,
+ "n": 9,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0000",
+ "l0008"
+ ],
+ "choice": "l0008"
+ },
+ {
+ "group_size": 2,
+ "n": 10,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0001",
+ "l0009"
+ ],
+ "choice": "l0009"
+ },
+ {
+ "group_size": 2,
+ "n": 11,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0002",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 2,
+ "n": 12,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 2,
+ "n": 13,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 2,
+ "n": 14,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 2,
+ "n": 15,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 2,
+ "n": 16,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 2,
+ "n": 17,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0000",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 2,
+ "n": 18,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0001",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 2,
+ "n": 19,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0002",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 20,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 21,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 22,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 23,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 24,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 25,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0008",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 26,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0009",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 27,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 28,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 29,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 30,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 31,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 32,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 33,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0000",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 34,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0001",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 35,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0002",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 36,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 37,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 38,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 39,
+ "rounds": [
+ [
+ 1,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 2,
+ "n": 40,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 2,
+ 2,
+ 2,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2,
+ 2
+ ],
+ [
+ 1,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 1,
+ "rounds": [],
+ "finalists": [
+ "l0000"
+ ],
+ "choice": "l0000"
+ },
+ {
+ "group_size": 3,
+ "n": 2,
+ "rounds": [],
+ "finalists": [
+ "l0000",
+ "l0001"
+ ],
+ "choice": "l0001"
+ },
+ {
+ "group_size": 3,
+ "n": 3,
+ "rounds": [],
+ "finalists": [
+ "l0000",
+ "l0001",
+ "l0002"
+ ],
+ "choice": "l0002"
+ },
+ {
+ "group_size": 3,
+ "n": 4,
+ "rounds": [
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0001",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 3,
+ "n": 5,
+ "rounds": [
+ [
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0001",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 3,
+ "n": 6,
+ "rounds": [
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0002",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 3,
+ "n": 7,
+ "rounds": [
+ [
+ 2,
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0001",
+ "l0003",
+ "l0006"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 3,
+ "n": 8,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0001",
+ "l0003",
+ "l0007"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 3,
+ "n": 9,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0002",
+ "l0003",
+ "l0008"
+ ],
+ "choice": "l0008"
+ },
+ {
+ "group_size": 3,
+ "n": 10,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 2,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0009"
+ ],
+ "choice": "l0009"
+ },
+ {
+ "group_size": 3,
+ "n": 11,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 3,
+ "n": 12,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 3,
+ "n": 13,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 2,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 3,
+ "n": 14,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 3,
+ "n": 15,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 3,
+ "n": 16,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 2,
+ 3,
+ 3
+ ],
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 3,
+ "n": 17,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 3,
+ "n": 18,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0008",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 3,
+ "n": 19,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 2,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0009",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 20,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 21,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 22,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 2,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 23,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 24,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 25,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 2,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 3,
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 26,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 3,
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 27,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 3,
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0008",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 28,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 2,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 2,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 29,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 2,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 30,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 2,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 31,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 32,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 33,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 34,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 35,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 36,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 2
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 37,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 2,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 38,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 2,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 39,
+ "rounds": [
+ [
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 2,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 3,
+ "n": 40,
+ "rounds": [
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 2,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3,
+ 3,
+ 3,
+ 3
+ ],
+ [
+ 2,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 1,
+ "rounds": [],
+ "finalists": [
+ "l0000"
+ ],
+ "choice": "l0000"
+ },
+ {
+ "group_size": 5,
+ "n": 2,
+ "rounds": [],
+ "finalists": [
+ "l0000",
+ "l0001"
+ ],
+ "choice": "l0001"
+ },
+ {
+ "group_size": 5,
+ "n": 3,
+ "rounds": [],
+ "finalists": [
+ "l0000",
+ "l0001",
+ "l0002"
+ ],
+ "choice": "l0002"
+ },
+ {
+ "group_size": 5,
+ "n": 4,
+ "rounds": [],
+ "finalists": [
+ "l0000",
+ "l0001",
+ "l0002",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 5,
+ "n": 5,
+ "rounds": [],
+ "finalists": [
+ "l0000",
+ "l0001",
+ "l0002",
+ "l0003",
+ "l0004"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 5,
+ "n": 6,
+ "rounds": [
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0002",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 5,
+ "n": 7,
+ "rounds": [
+ [
+ 3,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0002",
+ "l0003"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 5,
+ "n": 8,
+ "rounds": [
+ [
+ 4,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0007"
+ ],
+ "choice": "l0003"
+ },
+ {
+ "group_size": 5,
+ "n": 9,
+ "rounds": [
+ [
+ 4,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0008"
+ ],
+ "choice": "l0008"
+ },
+ {
+ "group_size": 5,
+ "n": 10,
+ "rounds": [
+ [
+ 5,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0009"
+ ],
+ "choice": "l0009"
+ },
+ {
+ "group_size": 5,
+ "n": 11,
+ "rounds": [
+ [
+ 3,
+ 4,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0002",
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 5,
+ "n": 12,
+ "rounds": [
+ [
+ 4,
+ 4,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0007",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 5,
+ "n": 13,
+ "rounds": [
+ [
+ 4,
+ 4,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0007",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 5,
+ "n": 14,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0008",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 5,
+ "n": 15,
+ "rounds": [
+ [
+ 5,
+ 5,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0009",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 5,
+ "n": 16,
+ "rounds": [
+ [
+ 4,
+ 4,
+ 4,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0007",
+ "l0010",
+ "l0012"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 5,
+ "n": 17,
+ "rounds": [
+ [
+ 4,
+ 4,
+ 4,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0007",
+ "l0010",
+ "l0012"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 5,
+ "n": 18,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 4,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0008",
+ "l0010",
+ "l0013"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 5,
+ "n": 19,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 5,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0008",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 20,
+ "rounds": [
+ [
+ 5,
+ 5,
+ 5,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0009",
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 21,
+ "rounds": [
+ [
+ 4,
+ 4,
+ 4,
+ 4,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0007",
+ "l0010",
+ "l0012",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 22,
+ "rounds": [
+ [
+ 4,
+ 4,
+ 5,
+ 4,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0007",
+ "l0010",
+ "l0013",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 23,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 4,
+ 5,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0008",
+ "l0010",
+ "l0013",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 24,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 5,
+ 5,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0008",
+ "l0010",
+ "l0018",
+ "l0019"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 25,
+ "rounds": [
+ [
+ 5,
+ 5,
+ 5,
+ 5,
+ 5
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0009",
+ "l0010",
+ "l0018",
+ "l0021"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 26,
+ "rounds": [
+ [
+ 4,
+ 4,
+ 5,
+ 4,
+ 4,
+ 5
+ ],
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 27,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 4,
+ 5,
+ 4,
+ 5
+ ],
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 28,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 5,
+ 4,
+ 5,
+ 5
+ ],
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 29,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5
+ ],
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 30,
+ "rounds": [
+ [
+ 5,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5
+ ],
+ [
+ 3,
+ 3
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 31,
+ "rounds": [
+ [
+ 4,
+ 4,
+ 5,
+ 4,
+ 5,
+ 4,
+ 5
+ ],
+ [
+ 3,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 32,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 4,
+ 5,
+ 4,
+ 5,
+ 5
+ ],
+ [
+ 3,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 33,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 5,
+ 4,
+ 5,
+ 5,
+ 5
+ ],
+ [
+ 3,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 34,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5
+ ],
+ [
+ 3,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 35,
+ "rounds": [
+ [
+ 5,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5
+ ],
+ [
+ 3,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 36,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 4,
+ 5,
+ 4,
+ 5,
+ 4,
+ 5
+ ],
+ [
+ 4,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 37,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 4,
+ 5,
+ 5,
+ 4,
+ 5,
+ 5
+ ],
+ [
+ 4,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 38,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 5,
+ 5,
+ 4,
+ 5,
+ 5,
+ 5
+ ],
+ [
+ 4,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0018",
+ "l0019"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 39,
+ "rounds": [
+ [
+ 4,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5
+ ],
+ [
+ 4,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0018",
+ "l0019"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 5,
+ "n": 40,
+ "rounds": [
+ [
+ 5,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5,
+ 5
+ ],
+ [
+ 4,
+ 4
+ ]
+ ],
+ "finalists": [
+ "l0018",
+ "l0038"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 16,
+ "n": 1,
+ "rounds": [],
+ "finalists": [
+ "l0000"
+ ],
+ "choice": "l0000"
+ },
+ {
+ "group_size": 16,
+ "n": 15,
+ "rounds": [],
+ "finalists": [
+ "l0000",
+ "l0001",
+ "l0002",
+ "l0003",
+ "l0004",
+ "l0005",
+ "l0006",
+ "l0007",
+ "l0008",
+ "l0009",
+ "l0010",
+ "l0011",
+ "l0012",
+ "l0013",
+ "l0014"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 16,
+ "n": 16,
+ "rounds": [],
+ "finalists": [
+ "l0000",
+ "l0001",
+ "l0002",
+ "l0003",
+ "l0004",
+ "l0005",
+ "l0006",
+ "l0007",
+ "l0008",
+ "l0009",
+ "l0010",
+ "l0011",
+ "l0012",
+ "l0013",
+ "l0014",
+ "l0015"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 16,
+ "n": 17,
+ "rounds": [
+ [
+ 8,
+ 9
+ ]
+ ],
+ "finalists": [
+ "l0003",
+ "l0010"
+ ],
+ "choice": "l0010"
+ },
+ {
+ "group_size": 16,
+ "n": 31,
+ "rounds": [
+ [
+ 15,
+ 16
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 16,
+ "n": 32,
+ "rounds": [
+ [
+ 16,
+ 16
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 16,
+ "n": 33,
+ "rounds": [
+ [
+ 11,
+ 11,
+ 11
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018",
+ "l0032"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 16,
+ "n": 77,
+ "rounds": [
+ [
+ 15,
+ 15,
+ 16,
+ 15,
+ 16
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018",
+ "l0038",
+ "l0054",
+ "l0072"
+ ],
+ "choice": "l0018"
+ },
+ {
+ "group_size": 16,
+ "n": 150,
+ "rounds": [
+ [
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018",
+ "l0038",
+ "l0054",
+ "l0072",
+ "l0083",
+ "l0102",
+ "l0106",
+ "l0128",
+ "l0146"
+ ],
+ "choice": "l0146"
+ },
+ {
+ "group_size": 16,
+ "n": 255,
+ "rounds": [
+ [
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018",
+ "l0038",
+ "l0054",
+ "l0078",
+ "l0090",
+ "l0102",
+ "l0124",
+ "l0142",
+ "l0146",
+ "l0160",
+ "l0177",
+ "l0201",
+ "l0222",
+ "l0230",
+ "l0239"
+ ],
+ "choice": "l0230"
+ },
+ {
+ "group_size": 16,
+ "n": 256,
+ "rounds": [
+ [
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16
+ ]
+ ],
+ "finalists": [
+ "l0010",
+ "l0018",
+ "l0038",
+ "l0054",
+ "l0078",
+ "l0090",
+ "l0102",
+ "l0124",
+ "l0142",
+ "l0146",
+ "l0160",
+ "l0177",
+ "l0201",
+ "l0223",
+ "l0230",
+ "l0252"
+ ],
+ "choice": "l0230"
+ },
+ {
+ "group_size": 16,
+ "n": 257,
+ "rounds": [
+ [
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 16,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 15,
+ 16
+ ],
+ [
+ 8,
+ 9
+ ]
+ ],
+ "finalists": [
+ "l0102",
+ "l0230"
+ ],
+ "choice": "l0230"
+ },
+ {
+ "group_size": 16,
+ "n": 300,
+ "rounds": [
+ [
+ 15,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16
+ ],
+ [
+ 9,
+ 10
+ ]
+ ],
+ "finalists": [
+ "l0140",
+ "l0230"
+ ],
+ "choice": "l0230"
+ },
+ {
+ "group_size": 16,
+ "n": 1000,
+ "rounds": [
+ [
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 15,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16,
+ 16
+ ],
+ [
+ 15,
+ 16,
+ 16,
+ 16
+ ]
+ ],
+ "finalists": [
+ "l0230",
+ "l0450",
+ "l0494",
+ "l0982"
+ ],
+ "choice": "l0450"
+ }
+ ],
+ "cases": [
+ {
+ "name": "default-group-77",
+ "group_size": null,
+ "questions": {
+ "q": {
+ "type": "choice",
+ "instructions": "pick",
+ "criteria": [
+ "l000",
+ "l001",
+ "l002",
+ "l003",
+ "l004",
+ "l005",
+ "l006",
+ "l007",
+ "l008",
+ "l009",
+ "l010",
+ "l011",
+ "l012",
+ "l013",
+ "l014",
+ "l015",
+ "l016",
+ "l017",
+ "l018",
+ "l019",
+ "l020",
+ "l021",
+ "l022",
+ "l023",
+ "l024",
+ "l025",
+ "l026",
+ "l027",
+ "l028",
+ "l029",
+ "l030",
+ "l031",
+ "l032",
+ "l033",
+ "l034",
+ "l035",
+ "l036",
+ "l037",
+ "l038",
+ "l039",
+ "l040",
+ "l041",
+ "l042",
+ "l043",
+ "l044",
+ "l045",
+ "l046",
+ "l047",
+ "l048",
+ "l049",
+ "l050",
+ "l051",
+ "l052",
+ "l053",
+ "l054",
+ "l055",
+ "l056",
+ "l057",
+ "l058",
+ "l059",
+ "l060",
+ "l061",
+ "l062",
+ "l063",
+ "l064",
+ "l065",
+ "l066",
+ "l067",
+ "l068",
+ "l069",
+ "l070",
+ "l071",
+ "l072",
+ "l073",
+ "l074",
+ "l075",
+ "l076"
+ ]
+ }
+ },
+ "calls": [
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l000",
+ "l001",
+ "l002",
+ "l003",
+ "l004",
+ "l005",
+ "l006",
+ "l007",
+ "l008",
+ "l009",
+ "l010",
+ "l011",
+ "l012",
+ "l013",
+ "l014"
+ ]
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l015",
+ "l016",
+ "l017",
+ "l018",
+ "l019",
+ "l020",
+ "l021",
+ "l022",
+ "l023",
+ "l024",
+ "l025",
+ "l026",
+ "l027",
+ "l028",
+ "l029"
+ ]
+ },
+ "2": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l030",
+ "l031",
+ "l032",
+ "l033",
+ "l034",
+ "l035",
+ "l036",
+ "l037",
+ "l038",
+ "l039",
+ "l040",
+ "l041",
+ "l042",
+ "l043",
+ "l044",
+ "l045"
+ ]
+ },
+ "3": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l046",
+ "l047",
+ "l048",
+ "l049",
+ "l050",
+ "l051",
+ "l052",
+ "l053",
+ "l054",
+ "l055",
+ "l056",
+ "l057",
+ "l058",
+ "l059",
+ "l060"
+ ]
+ },
+ "4": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l061",
+ "l062",
+ "l063",
+ "l064",
+ "l065",
+ "l066",
+ "l067",
+ "l068",
+ "l069",
+ "l070",
+ "l071",
+ "l072",
+ "l073",
+ "l074",
+ "l075",
+ "l076"
+ ]
+ }
+ },
+ {
+ "q": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l014",
+ "l023",
+ "l030",
+ "l060",
+ "l074"
+ ]
+ }
+ }
+ ],
+ "model": "stub-2",
+ "answers": {
+ "q": "l030"
+ },
+ "tournament": {
+ "q": {
+ "labels": [
+ "l014",
+ "l023",
+ "l030",
+ "l060",
+ "l074"
+ ],
+ "n": 77,
+ "rounds": 1
+ }
+ },
+ "caller_questions_unmutated": true
+ },
+ {
+ "name": "mixed-group-8",
+ "group_size": 8,
+ "questions": {
+ "big": {
+ "type": "choice",
+ "instructions": "pick",
+ "criteria": {
+ "l00": "about l00",
+ "l01": "about l01",
+ "l02": "about l02",
+ "l03": "about l03",
+ "l04": "about l04",
+ "l05": "about l05",
+ "l06": "about l06",
+ "l07": "about l07",
+ "l08": "about l08",
+ "l09": "about l09",
+ "l10": "about l10",
+ "l11": "about l11",
+ "l12": "about l12",
+ "l13": "about l13",
+ "l14": "about l14",
+ "l15": "about l15",
+ "l16": "about l16",
+ "l17": "about l17",
+ "l18": "about l18",
+ "l19": "about l19"
+ }
+ },
+ "small": {
+ "type": "choice",
+ "instructions": "pick",
+ "criteria": [
+ "x",
+ "y"
+ ]
+ },
+ "level": {
+ "type": "score",
+ "instructions": "rate",
+ "criteria": [
+ "low",
+ "high"
+ ]
+ },
+ "urgent": {
+ "type": "noul",
+ "instructions": "Is this urgent?"
+ }
+ },
+ "calls": [
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l00",
+ "l01",
+ "l02",
+ "l03",
+ "l04",
+ "l05"
+ ],
+ "criteria": {
+ "l00": "about l00",
+ "l01": "about l01",
+ "l02": "about l02",
+ "l03": "about l03",
+ "l04": "about l04",
+ "l05": "about l05"
+ }
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l06",
+ "l07",
+ "l08",
+ "l09",
+ "l10",
+ "l11",
+ "l12"
+ ],
+ "criteria": {
+ "l06": "about l06",
+ "l07": "about l07",
+ "l08": "about l08",
+ "l09": "about l09",
+ "l10": "about l10",
+ "l11": "about l11",
+ "l12": "about l12"
+ }
+ },
+ "2": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l13",
+ "l14",
+ "l15",
+ "l16",
+ "l17",
+ "l18",
+ "l19"
+ ],
+ "criteria": {
+ "l13": "about l13",
+ "l14": "about l14",
+ "l15": "about l15",
+ "l16": "about l16",
+ "l17": "about l17",
+ "l18": "about l18",
+ "l19": "about l19"
+ }
+ }
+ },
+ {
+ "big": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "l03",
+ "l10",
+ "l13"
+ ],
+ "criteria": {
+ "l03": "about l03",
+ "l10": "about l10",
+ "l13": "about l13"
+ }
+ },
+ "small": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "x",
+ "y"
+ ]
+ },
+ "level": {
+ "type": "score",
+ "instructions": "rate"
+ },
+ "urgent": {
+ "type": "noul",
+ "instructions": "Is this urgent?"
+ }
+ }
+ ],
+ "model": "stub-2",
+ "answers": {
+ "big": "l10",
+ "small": "y"
+ },
+ "tournament": {
+ "big": {
+ "labels": [
+ "l03",
+ "l10",
+ "l13"
+ ],
+ "n": 20,
+ "rounds": 1
+ },
+ "small": {
+ "labels": [
+ "x",
+ "y"
+ ],
+ "n": 2,
+ "rounds": 0
+ }
+ },
+ "caller_questions_unmutated": true
+ },
+ {
+ "name": "nothing-to-narrow",
+ "group_size": null,
+ "questions": {
+ "small": {
+ "type": "choice",
+ "instructions": "pick",
+ "criteria": [
+ "x",
+ "y"
+ ]
+ },
+ "level": {
+ "type": "score",
+ "instructions": "rate",
+ "criteria": [
+ "low",
+ "high"
+ ]
+ }
+ },
+ "calls": [
+ {
+ "small": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "x",
+ "y"
+ ]
+ },
+ "level": {
+ "type": "score",
+ "instructions": "rate"
+ }
+ }
+ ],
+ "model": "stub-1",
+ "answers": {
+ "small": "y"
+ },
+ "tournament": {
+ "small": {
+ "labels": [
+ "x",
+ "y"
+ ],
+ "n": 2,
+ "rounds": 0
+ }
+ },
+ "caller_questions_unmutated": true
+ },
+ {
+ "name": "two-choices-different-rounds",
+ "group_size": null,
+ "questions": {
+ "wide": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "criteria": [
+ "w000",
+ "w001",
+ "w002",
+ "w003",
+ "w004",
+ "w005",
+ "w006",
+ "w007",
+ "w008",
+ "w009",
+ "w010",
+ "w011",
+ "w012",
+ "w013",
+ "w014",
+ "w015",
+ "w016",
+ "w017",
+ "w018",
+ "w019",
+ "w020",
+ "w021",
+ "w022",
+ "w023",
+ "w024",
+ "w025",
+ "w026",
+ "w027",
+ "w028",
+ "w029",
+ "w030",
+ "w031",
+ "w032",
+ "w033",
+ "w034",
+ "w035",
+ "w036",
+ "w037",
+ "w038",
+ "w039",
+ "w040",
+ "w041",
+ "w042",
+ "w043",
+ "w044",
+ "w045",
+ "w046",
+ "w047",
+ "w048",
+ "w049",
+ "w050",
+ "w051",
+ "w052",
+ "w053",
+ "w054",
+ "w055",
+ "w056",
+ "w057",
+ "w058",
+ "w059",
+ "w060",
+ "w061",
+ "w062",
+ "w063",
+ "w064",
+ "w065",
+ "w066",
+ "w067",
+ "w068",
+ "w069",
+ "w070",
+ "w071",
+ "w072",
+ "w073",
+ "w074",
+ "w075",
+ "w076",
+ "w077",
+ "w078",
+ "w079",
+ "w080",
+ "w081",
+ "w082",
+ "w083",
+ "w084",
+ "w085",
+ "w086",
+ "w087",
+ "w088",
+ "w089",
+ "w090",
+ "w091",
+ "w092",
+ "w093",
+ "w094",
+ "w095",
+ "w096",
+ "w097",
+ "w098",
+ "w099",
+ "w100",
+ "w101",
+ "w102",
+ "w103",
+ "w104",
+ "w105",
+ "w106",
+ "w107",
+ "w108",
+ "w109",
+ "w110",
+ "w111",
+ "w112",
+ "w113",
+ "w114",
+ "w115",
+ "w116",
+ "w117",
+ "w118",
+ "w119",
+ "w120",
+ "w121",
+ "w122",
+ "w123",
+ "w124",
+ "w125",
+ "w126",
+ "w127",
+ "w128",
+ "w129",
+ "w130",
+ "w131",
+ "w132",
+ "w133",
+ "w134",
+ "w135",
+ "w136",
+ "w137",
+ "w138",
+ "w139",
+ "w140",
+ "w141",
+ "w142",
+ "w143",
+ "w144",
+ "w145",
+ "w146",
+ "w147",
+ "w148",
+ "w149",
+ "w150",
+ "w151",
+ "w152",
+ "w153",
+ "w154",
+ "w155",
+ "w156",
+ "w157",
+ "w158",
+ "w159",
+ "w160",
+ "w161",
+ "w162",
+ "w163",
+ "w164",
+ "w165",
+ "w166",
+ "w167",
+ "w168",
+ "w169",
+ "w170",
+ "w171",
+ "w172",
+ "w173",
+ "w174",
+ "w175",
+ "w176",
+ "w177",
+ "w178",
+ "w179",
+ "w180",
+ "w181",
+ "w182",
+ "w183",
+ "w184",
+ "w185",
+ "w186",
+ "w187",
+ "w188",
+ "w189",
+ "w190",
+ "w191",
+ "w192",
+ "w193",
+ "w194",
+ "w195",
+ "w196",
+ "w197",
+ "w198",
+ "w199",
+ "w200",
+ "w201",
+ "w202",
+ "w203",
+ "w204",
+ "w205",
+ "w206",
+ "w207",
+ "w208",
+ "w209",
+ "w210",
+ "w211",
+ "w212",
+ "w213",
+ "w214",
+ "w215",
+ "w216",
+ "w217",
+ "w218",
+ "w219",
+ "w220",
+ "w221",
+ "w222",
+ "w223",
+ "w224",
+ "w225",
+ "w226",
+ "w227",
+ "w228",
+ "w229",
+ "w230",
+ "w231",
+ "w232",
+ "w233",
+ "w234",
+ "w235",
+ "w236",
+ "w237",
+ "w238",
+ "w239",
+ "w240",
+ "w241",
+ "w242",
+ "w243",
+ "w244",
+ "w245",
+ "w246",
+ "w247",
+ "w248",
+ "w249",
+ "w250",
+ "w251",
+ "w252",
+ "w253",
+ "w254",
+ "w255",
+ "w256",
+ "w257",
+ "w258",
+ "w259",
+ "w260",
+ "w261",
+ "w262",
+ "w263",
+ "w264",
+ "w265",
+ "w266",
+ "w267",
+ "w268",
+ "w269",
+ "w270",
+ "w271",
+ "w272",
+ "w273",
+ "w274",
+ "w275",
+ "w276",
+ "w277",
+ "w278",
+ "w279",
+ "w280",
+ "w281",
+ "w282",
+ "w283",
+ "w284",
+ "w285",
+ "w286",
+ "w287",
+ "w288",
+ "w289",
+ "w290",
+ "w291",
+ "w292",
+ "w293",
+ "w294",
+ "w295",
+ "w296",
+ "w297",
+ "w298",
+ "w299"
+ ]
+ },
+ "narrow": {
+ "type": "choice",
+ "instructions": "Which other?",
+ "criteria": [
+ "n00",
+ "n01",
+ "n02",
+ "n03",
+ "n04",
+ "n05",
+ "n06",
+ "n07",
+ "n08",
+ "n09",
+ "n10",
+ "n11",
+ "n12",
+ "n13",
+ "n14",
+ "n15",
+ "n16",
+ "n17",
+ "n18",
+ "n19",
+ "n20",
+ "n21",
+ "n22",
+ "n23",
+ "n24",
+ "n25",
+ "n26",
+ "n27",
+ "n28",
+ "n29",
+ "n30",
+ "n31",
+ "n32",
+ "n33",
+ "n34",
+ "n35",
+ "n36",
+ "n37",
+ "n38",
+ "n39"
+ ]
+ }
+ },
+ "calls": [
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w000",
+ "w001",
+ "w002",
+ "w003",
+ "w004",
+ "w005",
+ "w006",
+ "w007",
+ "w008",
+ "w009",
+ "w010",
+ "w011",
+ "w012",
+ "w013",
+ "w014"
+ ]
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w015",
+ "w016",
+ "w017",
+ "w018",
+ "w019",
+ "w020",
+ "w021",
+ "w022",
+ "w023",
+ "w024",
+ "w025",
+ "w026",
+ "w027",
+ "w028",
+ "w029",
+ "w030"
+ ]
+ },
+ "2": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w031",
+ "w032",
+ "w033",
+ "w034",
+ "w035",
+ "w036",
+ "w037",
+ "w038",
+ "w039",
+ "w040",
+ "w041",
+ "w042",
+ "w043",
+ "w044",
+ "w045",
+ "w046"
+ ]
+ },
+ "3": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w047",
+ "w048",
+ "w049",
+ "w050",
+ "w051",
+ "w052",
+ "w053",
+ "w054",
+ "w055",
+ "w056",
+ "w057",
+ "w058",
+ "w059",
+ "w060",
+ "w061",
+ "w062"
+ ]
+ },
+ "4": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w063",
+ "w064",
+ "w065",
+ "w066",
+ "w067",
+ "w068",
+ "w069",
+ "w070",
+ "w071",
+ "w072",
+ "w073",
+ "w074",
+ "w075",
+ "w076",
+ "w077"
+ ]
+ },
+ "5": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w078",
+ "w079",
+ "w080",
+ "w081",
+ "w082",
+ "w083",
+ "w084",
+ "w085",
+ "w086",
+ "w087",
+ "w088",
+ "w089",
+ "w090",
+ "w091",
+ "w092",
+ "w093"
+ ]
+ },
+ "6": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w094",
+ "w095",
+ "w096",
+ "w097",
+ "w098",
+ "w099",
+ "w100",
+ "w101",
+ "w102",
+ "w103",
+ "w104",
+ "w105",
+ "w106",
+ "w107",
+ "w108",
+ "w109"
+ ]
+ },
+ "7": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w110",
+ "w111",
+ "w112",
+ "w113",
+ "w114",
+ "w115",
+ "w116",
+ "w117",
+ "w118",
+ "w119",
+ "w120",
+ "w121",
+ "w122",
+ "w123",
+ "w124",
+ "w125"
+ ]
+ },
+ "8": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w126",
+ "w127",
+ "w128",
+ "w129",
+ "w130",
+ "w131",
+ "w132",
+ "w133",
+ "w134",
+ "w135",
+ "w136",
+ "w137",
+ "w138",
+ "w139",
+ "w140",
+ "w141"
+ ]
+ },
+ "9": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w142",
+ "w143",
+ "w144",
+ "w145",
+ "w146",
+ "w147",
+ "w148",
+ "w149",
+ "w150",
+ "w151",
+ "w152",
+ "w153",
+ "w154",
+ "w155",
+ "w156"
+ ]
+ },
+ "10": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w157",
+ "w158",
+ "w159",
+ "w160",
+ "w161",
+ "w162",
+ "w163",
+ "w164",
+ "w165",
+ "w166",
+ "w167",
+ "w168",
+ "w169",
+ "w170",
+ "w171",
+ "w172"
+ ]
+ },
+ "11": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w173",
+ "w174",
+ "w175",
+ "w176",
+ "w177",
+ "w178",
+ "w179",
+ "w180",
+ "w181",
+ "w182",
+ "w183",
+ "w184",
+ "w185",
+ "w186",
+ "w187",
+ "w188"
+ ]
+ },
+ "12": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w189",
+ "w190",
+ "w191",
+ "w192",
+ "w193",
+ "w194",
+ "w195",
+ "w196",
+ "w197",
+ "w198",
+ "w199",
+ "w200",
+ "w201",
+ "w202",
+ "w203",
+ "w204"
+ ]
+ },
+ "13": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w205",
+ "w206",
+ "w207",
+ "w208",
+ "w209",
+ "w210",
+ "w211",
+ "w212",
+ "w213",
+ "w214",
+ "w215",
+ "w216",
+ "w217",
+ "w218",
+ "w219",
+ "w220"
+ ]
+ },
+ "14": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w221",
+ "w222",
+ "w223",
+ "w224",
+ "w225",
+ "w226",
+ "w227",
+ "w228",
+ "w229",
+ "w230",
+ "w231",
+ "w232",
+ "w233",
+ "w234",
+ "w235"
+ ]
+ },
+ "15": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w236",
+ "w237",
+ "w238",
+ "w239",
+ "w240",
+ "w241",
+ "w242",
+ "w243",
+ "w244",
+ "w245",
+ "w246",
+ "w247",
+ "w248",
+ "w249",
+ "w250",
+ "w251"
+ ]
+ },
+ "16": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w252",
+ "w253",
+ "w254",
+ "w255",
+ "w256",
+ "w257",
+ "w258",
+ "w259",
+ "w260",
+ "w261",
+ "w262",
+ "w263",
+ "w264",
+ "w265",
+ "w266",
+ "w267"
+ ]
+ },
+ "17": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w268",
+ "w269",
+ "w270",
+ "w271",
+ "w272",
+ "w273",
+ "w274",
+ "w275",
+ "w276",
+ "w277",
+ "w278",
+ "w279",
+ "w280",
+ "w281",
+ "w282",
+ "w283"
+ ]
+ },
+ "18": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w284",
+ "w285",
+ "w286",
+ "w287",
+ "w288",
+ "w289",
+ "w290",
+ "w291",
+ "w292",
+ "w293",
+ "w294",
+ "w295",
+ "w296",
+ "w297",
+ "w298",
+ "w299"
+ ]
+ },
+ "19": {
+ "type": "choice",
+ "instructions": "Which other?",
+ "labels": [
+ "n00",
+ "n01",
+ "n02",
+ "n03",
+ "n04",
+ "n05",
+ "n06",
+ "n07",
+ "n08",
+ "n09",
+ "n10",
+ "n11",
+ "n12"
+ ]
+ },
+ "20": {
+ "type": "choice",
+ "instructions": "Which other?",
+ "labels": [
+ "n13",
+ "n14",
+ "n15",
+ "n16",
+ "n17",
+ "n18",
+ "n19",
+ "n20",
+ "n21",
+ "n22",
+ "n23",
+ "n24",
+ "n25"
+ ]
+ },
+ "21": {
+ "type": "choice",
+ "instructions": "Which other?",
+ "labels": [
+ "n26",
+ "n27",
+ "n28",
+ "n29",
+ "n30",
+ "n31",
+ "n32",
+ "n33",
+ "n34",
+ "n35",
+ "n36",
+ "n37",
+ "n38",
+ "n39"
+ ]
+ }
+ },
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w000",
+ "w017",
+ "w039",
+ "w048",
+ "w063",
+ "w079",
+ "w099",
+ "w123",
+ "w130"
+ ]
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w152",
+ "w170",
+ "w178",
+ "w198",
+ "w220",
+ "w228",
+ "w246",
+ "w260",
+ "w268",
+ "w288"
+ ]
+ }
+ },
+ {
+ "wide": {
+ "type": "choice",
+ "instructions": "Which one?",
+ "labels": [
+ "w000",
+ "w178"
+ ]
+ },
+ "narrow": {
+ "type": "choice",
+ "instructions": "Which other?",
+ "labels": [
+ "n12",
+ "n23",
+ "n38"
+ ]
+ }
+ }
+ ],
+ "model": "stub-3",
+ "answers": {
+ "wide": "w000",
+ "narrow": "n38"
+ },
+ "tournament": {
+ "wide": {
+ "labels": [
+ "w000",
+ "w178"
+ ],
+ "n": 300,
+ "rounds": 2
+ },
+ "narrow": {
+ "labels": [
+ "n12",
+ "n23",
+ "n38"
+ ],
+ "n": 40,
+ "rounds": 1
+ }
+ },
+ "caller_questions_unmutated": true
+ },
+ {
+ "name": "described-criteria",
+ "group_size": 3,
+ "questions": {
+ "intent": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "criteria": {
+ "card_arrival": null,
+ "card_delivery_estimate": "",
+ "card_not_working": 2,
+ "cash_withdrawal_charge": "cash withdrawal charge",
+ "declined_card_payment": null,
+ "direct_debit_payment_not_recognised": "",
+ "exchange_rate": 6,
+ "failed_transfer": "failed transfer",
+ "lost_or_stolen_card": null,
+ "pending_card_payment": "",
+ "refund_not_showing_up": 10,
+ "request_refund": "request refund",
+ "reverted_card_payment": null,
+ "top_up_failed": "",
+ "transfer_fee_charged": 14,
+ "verify_my_identity": "verify my identity",
+ "wrong_amount_of_cash_received": null
+ }
+ }
+ },
+ "calls": [
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "card_arrival",
+ "card_delivery_estimate"
+ ],
+ "criteria": {
+ "card_arrival": null,
+ "card_delivery_estimate": ""
+ }
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "card_not_working",
+ "cash_withdrawal_charge",
+ "declined_card_payment"
+ ],
+ "criteria": {
+ "card_not_working": 2,
+ "cash_withdrawal_charge": "cash withdrawal charge",
+ "declined_card_payment": null
+ }
+ },
+ "2": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "direct_debit_payment_not_recognised",
+ "exchange_rate",
+ "failed_transfer"
+ ],
+ "criteria": {
+ "direct_debit_payment_not_recognised": "",
+ "exchange_rate": 6,
+ "failed_transfer": "failed transfer"
+ }
+ },
+ "3": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "lost_or_stolen_card",
+ "pending_card_payment",
+ "refund_not_showing_up"
+ ],
+ "criteria": {
+ "lost_or_stolen_card": null,
+ "pending_card_payment": "",
+ "refund_not_showing_up": 10
+ }
+ },
+ "4": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "request_refund",
+ "reverted_card_payment",
+ "top_up_failed"
+ ],
+ "criteria": {
+ "request_refund": "request refund",
+ "reverted_card_payment": null,
+ "top_up_failed": ""
+ }
+ },
+ "5": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "transfer_fee_charged",
+ "verify_my_identity",
+ "wrong_amount_of_cash_received"
+ ],
+ "criteria": {
+ "transfer_fee_charged": 14,
+ "verify_my_identity": "verify my identity",
+ "wrong_amount_of_cash_received": null
+ }
+ }
+ },
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "card_delivery_estimate",
+ "card_not_working",
+ "direct_debit_payment_not_recognised"
+ ],
+ "criteria": {
+ "card_delivery_estimate": "",
+ "card_not_working": 2,
+ "direct_debit_payment_not_recognised": ""
+ }
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "refund_not_showing_up",
+ "top_up_failed",
+ "wrong_amount_of_cash_received"
+ ],
+ "criteria": {
+ "refund_not_showing_up": 10,
+ "top_up_failed": "",
+ "wrong_amount_of_cash_received": null
+ }
+ }
+ },
+ {
+ "intent": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "direct_debit_payment_not_recognised",
+ "top_up_failed"
+ ],
+ "criteria": {
+ "direct_debit_payment_not_recognised": "",
+ "top_up_failed": ""
+ }
+ }
+ }
+ ],
+ "model": "stub-3",
+ "answers": {
+ "intent": "top_up_failed"
+ },
+ "tournament": {
+ "intent": {
+ "labels": [
+ "direct_debit_payment_not_recognised",
+ "top_up_failed"
+ ],
+ "n": 17,
+ "rounds": 2
+ }
+ },
+ "caller_questions_unmutated": true
+ },
+ {
+ "name": "group-of-one",
+ "group_size": 2,
+ "questions": {
+ "q": {
+ "type": "choice",
+ "instructions": "pick",
+ "criteria": [
+ "e",
+ "d",
+ "c",
+ "b",
+ "a"
+ ]
+ }
+ },
+ "calls": [
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "e"
+ ]
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "d",
+ "c"
+ ]
+ },
+ "2": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "b",
+ "a"
+ ]
+ }
+ },
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "e"
+ ]
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "d",
+ "b"
+ ]
+ }
+ },
+ {
+ "q": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "e",
+ "d"
+ ]
+ }
+ }
+ ],
+ "model": "stub-3",
+ "answers": {
+ "q": "e"
+ },
+ "tournament": {
+ "q": {
+ "labels": [
+ "e",
+ "d"
+ ],
+ "n": 5,
+ "rounds": 2
+ }
+ },
+ "caller_questions_unmutated": true
+ },
+ {
+ "name": "criteria-order-not-sorted",
+ "group_size": 4,
+ "questions": {
+ "q": {
+ "type": "choice",
+ "instructions": "pick",
+ "criteria": [
+ "wrong_amount_of_cash_received",
+ "verify_my_identity",
+ "transfer_fee_charged",
+ "top_up_failed",
+ "reverted_card_payment",
+ "request_refund",
+ "refund_not_showing_up",
+ "pending_card_payment",
+ "lost_or_stolen_card",
+ "failed_transfer",
+ "exchange_rate",
+ "direct_debit_payment_not_recognised",
+ "declined_card_payment",
+ "cash_withdrawal_charge",
+ "card_not_working",
+ "card_delivery_estimate",
+ "card_arrival"
+ ]
+ }
+ },
+ "calls": [
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "wrong_amount_of_cash_received",
+ "verify_my_identity",
+ "transfer_fee_charged"
+ ]
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "top_up_failed",
+ "reverted_card_payment",
+ "request_refund"
+ ]
+ },
+ "2": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "refund_not_showing_up",
+ "pending_card_payment",
+ "lost_or_stolen_card",
+ "failed_transfer"
+ ]
+ },
+ "3": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "exchange_rate",
+ "direct_debit_payment_not_recognised",
+ "declined_card_payment"
+ ]
+ },
+ "4": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "cash_withdrawal_charge",
+ "card_not_working",
+ "card_delivery_estimate",
+ "card_arrival"
+ ]
+ }
+ },
+ {
+ "0": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "wrong_amount_of_cash_received",
+ "top_up_failed"
+ ]
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "refund_not_showing_up",
+ "direct_debit_payment_not_recognised",
+ "card_not_working"
+ ]
+ }
+ },
+ {
+ "q": {
+ "type": "choice",
+ "instructions": "pick",
+ "labels": [
+ "top_up_failed",
+ "direct_debit_payment_not_recognised"
+ ]
+ }
+ }
+ ],
+ "model": "stub-3",
+ "answers": {
+ "q": "top_up_failed"
+ },
+ "tournament": {
+ "q": {
+ "labels": [
+ "top_up_failed",
+ "direct_debit_payment_not_recognised"
+ ],
+ "n": 17,
+ "rounds": 2
+ }
+ },
+ "caller_questions_unmutated": true
+ }
+ ],
+ "refusals": [
+ {
+ "group_size": 1,
+ "error": "ValueError",
+ "message": "group_size must be an integer of at least 2, got 1"
+ },
+ {
+ "group_size": 0,
+ "error": "ValueError",
+ "message": "group_size must be an integer of at least 2, got 0"
+ },
+ {
+ "group_size": -3,
+ "error": "ValueError",
+ "message": "group_size must be an integer of at least 2, got -3"
+ }
+ ]
+}
diff --git a/laya-java/fixtures/tournament_golden.json b/laya-java/fixtures/tournament_golden.json
new file mode 100644
index 00000000..69cdc1d5
--- /dev/null
+++ b/laya-java/fixtures/tournament_golden.json
@@ -0,0 +1,509 @@
+{
+ "checkpoint": "multilingual",
+ "graph": "laya.onnx",
+ "cases": [
+ {
+ "name": "banking-default-group",
+ "state": "My new card still has not arrived after two weeks.",
+ "group_size": null,
+ "questions": {
+ "intent": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "criteria": [
+ "activate_my_card",
+ "age_limit",
+ "apple_pay_or_google_pay",
+ "atm_support",
+ "automatic_top_up",
+ "balance_not_updated_after_bank_transfer",
+ "beneficiary_not_allowed",
+ "cancel_transfer",
+ "card_about_to_expire",
+ "card_acceptance",
+ "card_arrival",
+ "card_delivery_estimate",
+ "card_linking",
+ "card_not_working",
+ "card_payment_fee_charged",
+ "card_payment_not_recognised",
+ "card_swallowed",
+ "cash_withdrawal_charge",
+ "change_pin",
+ "compromised_card",
+ "contactless_not_working",
+ "country_support",
+ "declined_card_payment",
+ "declined_cash_withdrawal",
+ "declined_transfer",
+ "edit_personal_details",
+ "exchange_charge",
+ "exchange_rate",
+ "failed_transfer",
+ "fiat_currency_support",
+ "get_physical_card",
+ "lost_or_stolen_card",
+ "lost_or_stolen_phone",
+ "order_physical_card",
+ "passcode_forgotten",
+ "pending_card_payment",
+ "refund_not_showing_up",
+ "request_refund",
+ "terminate_account",
+ "top_up_failed"
+ ]
+ }
+ },
+ "calls": [
+ {
+ "asked": {
+ "0": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "activate_my_card",
+ "age_limit",
+ "apple_pay_or_google_pay",
+ "atm_support",
+ "automatic_top_up",
+ "balance_not_updated_after_bank_transfer",
+ "beneficiary_not_allowed",
+ "cancel_transfer",
+ "card_about_to_expire",
+ "card_acceptance",
+ "card_arrival",
+ "card_delivery_estimate",
+ "card_linking"
+ ]
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "card_not_working",
+ "card_payment_fee_charged",
+ "card_payment_not_recognised",
+ "card_swallowed",
+ "cash_withdrawal_charge",
+ "change_pin",
+ "compromised_card",
+ "contactless_not_working",
+ "country_support",
+ "declined_card_payment",
+ "declined_cash_withdrawal",
+ "declined_transfer",
+ "edit_personal_details"
+ ]
+ },
+ "2": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "exchange_charge",
+ "exchange_rate",
+ "failed_transfer",
+ "fiat_currency_support",
+ "get_physical_card",
+ "lost_or_stolen_card",
+ "lost_or_stolen_phone",
+ "order_physical_card",
+ "passcode_forgotten",
+ "pending_card_payment",
+ "refund_not_showing_up",
+ "request_refund",
+ "terminate_account",
+ "top_up_failed"
+ ]
+ }
+ },
+ "answers": {
+ "0": {
+ "type": "choice",
+ "choice": "card_about_to_expire",
+ "probabilities": {
+ "activate_my_card": 0.0049,
+ "age_limit": 0.0001,
+ "apple_pay_or_google_pay": 0.0001,
+ "atm_support": 0.0001,
+ "automatic_top_up": 0.0002,
+ "balance_not_updated_after_bank_transfer": 0.0004,
+ "beneficiary_not_allowed": 0.0001,
+ "cancel_transfer": 0.0001,
+ "card_about_to_expire": 0.8734,
+ "card_acceptance": 0.0036,
+ "card_arrival": 0.1124,
+ "card_delivery_estimate": 0.0041,
+ "card_linking": 0.0006
+ },
+ "confidence": 0.8264,
+ "answer_confidence": 0.8734,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "1": {
+ "type": "choice",
+ "choice": "card_not_working",
+ "probabilities": {
+ "card_not_working": 0.9967,
+ "card_payment_fee_charged": 0.0,
+ "card_payment_not_recognised": 0.0001,
+ "card_swallowed": 0.0009,
+ "cash_withdrawal_charge": 0.0,
+ "change_pin": 0.0,
+ "compromised_card": 0.0016,
+ "contactless_not_working": 0.0005,
+ "country_support": 0.0,
+ "declined_card_payment": 0.0,
+ "declined_cash_withdrawal": 0.0,
+ "declined_transfer": 0.0,
+ "edit_personal_details": 0.0
+ },
+ "confidence": 0.9898,
+ "answer_confidence": 0.9967,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "2": {
+ "type": "choice",
+ "choice": "lost_or_stolen_card",
+ "probabilities": {
+ "exchange_charge": 0.0001,
+ "exchange_rate": 0.0002,
+ "failed_transfer": 0.0929,
+ "fiat_currency_support": 0.0003,
+ "get_physical_card": 0.0002,
+ "lost_or_stolen_card": 0.7004,
+ "lost_or_stolen_phone": 0.017,
+ "order_physical_card": 0.003,
+ "passcode_forgotten": 0.016,
+ "pending_card_payment": 0.1542,
+ "refund_not_showing_up": 0.0015,
+ "request_refund": 0.0001,
+ "terminate_account": 0.0027,
+ "top_up_failed": 0.0116
+ },
+ "confidence": 0.6228,
+ "answer_confidence": 0.7004,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 303,
+ "output_tokens": 0,
+ "state_tokens": 11,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "asked": {
+ "intent": {
+ "type": "choice",
+ "instructions": "What does the customer want?",
+ "labels": [
+ "card_about_to_expire",
+ "card_not_working",
+ "lost_or_stolen_card"
+ ]
+ }
+ },
+ "answers": {
+ "intent": {
+ "type": "choice",
+ "choice": "card_about_to_expire",
+ "probabilities": {
+ "card_about_to_expire": 0.9919,
+ "card_not_working": 0.0032,
+ "lost_or_stolen_card": 0.0049
+ },
+ "confidence": 0.9523,
+ "answer_confidence": 0.9919,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 46,
+ "output_tokens": 0,
+ "state_tokens": 11,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "model": "laya-rl-agent-onnx",
+ "tournament": {
+ "intent": {
+ "labels": [
+ "card_about_to_expire",
+ "card_not_working",
+ "lost_or_stolen_card"
+ ],
+ "n": 40,
+ "rounds": 1
+ }
+ }
+ },
+ {
+ "name": "mixed-group-8",
+ "state": "We were billed twice for March and want a refund today.",
+ "group_size": 8,
+ "questions": {
+ "dept": {
+ "type": "choice",
+ "instructions": "Which team owns this?",
+ "criteria": {
+ "billing": "invoices and charges",
+ "refunds": "money sent back",
+ "fraud": "unauthorised activity",
+ "cards": "card problems",
+ "transfers": "bank transfers",
+ "loans": "credit and loans",
+ "savings": "savings products",
+ "kyc": "identity checks",
+ "support": "general help",
+ "legal": "complaints and law",
+ "sales": "new products",
+ "tech": "app bugs"
+ }
+ },
+ "urgent": {
+ "type": "noul",
+ "instructions": "This needs a human today."
+ }
+ },
+ "calls": [
+ {
+ "asked": {
+ "0": {
+ "type": "choice",
+ "instructions": "Which team owns this?",
+ "labels": [
+ "billing",
+ "refunds",
+ "fraud",
+ "cards",
+ "transfers",
+ "loans"
+ ],
+ "criteria": {
+ "billing": "invoices and charges",
+ "refunds": "money sent back",
+ "fraud": "unauthorised activity",
+ "cards": "card problems",
+ "transfers": "bank transfers",
+ "loans": "credit and loans"
+ }
+ },
+ "1": {
+ "type": "choice",
+ "instructions": "Which team owns this?",
+ "labels": [
+ "savings",
+ "kyc",
+ "support",
+ "legal",
+ "sales",
+ "tech"
+ ],
+ "criteria": {
+ "savings": "savings products",
+ "kyc": "identity checks",
+ "support": "general help",
+ "legal": "complaints and law",
+ "sales": "new products",
+ "tech": "app bugs"
+ }
+ }
+ },
+ "answers": {
+ "0": {
+ "type": "choice",
+ "choice": "refunds",
+ "probabilities": {
+ "billing": 0.0011,
+ "refunds": 0.9988,
+ "fraud": 0.0,
+ "cards": 0.0,
+ "transfers": 0.0,
+ "loans": 0.0
+ },
+ "confidence": 0.9949,
+ "answer_confidence": 0.9988,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "1": {
+ "type": "choice",
+ "choice": "sales",
+ "probabilities": {
+ "savings": 0.3114,
+ "kyc": 0.1803,
+ "support": 0.0423,
+ "legal": 0.0523,
+ "sales": 0.3787,
+ "tech": 0.0351
+ },
+ "confidence": 0.1933,
+ "answer_confidence": 0.3787,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 113,
+ "output_tokens": 0,
+ "state_tokens": 12,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ },
+ {
+ "asked": {
+ "dept": {
+ "type": "choice",
+ "instructions": "Which team owns this?",
+ "labels": [
+ "refunds",
+ "sales"
+ ],
+ "criteria": {
+ "refunds": "money sent back",
+ "sales": "new products"
+ }
+ },
+ "urgent": {
+ "type": "noul",
+ "instructions": "This needs a human today."
+ }
+ },
+ "answers": {
+ "dept": {
+ "type": "choice",
+ "choice": "refunds",
+ "probabilities": {
+ "refunds": 1.0,
+ "sales": 0.0
+ },
+ "confidence": 0.9997,
+ "answer_confidence": 1.0,
+ "action": {
+ "act_probability": 1.0
+ }
+ },
+ "urgent": {
+ "type": "noul",
+ "noul": 0.0283,
+ "confidence": 0.9717,
+ "answer_confidence": 0.9717,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 79,
+ "output_tokens": 0,
+ "state_tokens": 12,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "model": "laya-rl-agent-onnx",
+ "tournament": {
+ "dept": {
+ "labels": [
+ "refunds",
+ "sales"
+ ],
+ "n": 12,
+ "rounds": 1
+ }
+ }
+ },
+ {
+ "name": "nothing-to-narrow",
+ "state": "The app crashes when I open statements.",
+ "group_size": null,
+ "questions": {
+ "intent": {
+ "type": "choice",
+ "instructions": "What is wrong?",
+ "criteria": {
+ "bug": "the app fails",
+ "billing": "a charge",
+ "other": "else"
+ }
+ }
+ },
+ "calls": [
+ {
+ "asked": {
+ "intent": {
+ "type": "choice",
+ "instructions": "What is wrong?",
+ "labels": [
+ "bug",
+ "billing",
+ "other"
+ ],
+ "criteria": {
+ "bug": "the app fails",
+ "billing": "a charge",
+ "other": "else"
+ }
+ }
+ },
+ "answers": {
+ "intent": {
+ "type": "choice",
+ "choice": "bug",
+ "probabilities": {
+ "bug": 0.9925,
+ "billing": 0.0015,
+ "other": 0.006
+ },
+ "confidence": 0.9563,
+ "answer_confidence": 0.9925,
+ "action": {
+ "act_probability": 1.0
+ }
+ }
+ },
+ "usage": {
+ "input_tokens": 34,
+ "output_tokens": 0,
+ "state_tokens": 8,
+ "state_tokens_dropped": 0,
+ "truncated": false,
+ "truncated_questions": []
+ }
+ }
+ ],
+ "model": "laya-rl-agent-onnx",
+ "tournament": {
+ "intent": {
+ "labels": [
+ "bug",
+ "billing",
+ "other"
+ ],
+ "n": 3,
+ "rounds": 0
+ }
+ }
+ }
+ ]
+}
diff --git a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Shortlist.java b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Shortlist.java
index f4d1ee8c..670bf6e4 100644
--- a/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Shortlist.java
+++ b/laya-java/laya-java/src/main/java/com/convaiinnovations/laya/Shortlist.java
@@ -13,7 +13,7 @@ import java.util.Set;
import java.util.concurrent.locks.ReentrantLock;
/**
- * Opt-in embedding shortlist for a choice question with many labels. A port of
+ * Opt-in embedding shortlist and tournament for a choice question with many labels. A port of
* {@code laya.shortlist}.
*
* <p>Every option of a choice shares one {@code head_max_len} budget, so a large label set leaves
@@ -42,6 +42,9 @@ import java.util.concurrent.locks.ReentrantLock;
* paying per embedding call cares about that last part.
* </ul>
*
+ * <p>{@link #predictTournament} needs no embedder: the decision model answers the labels in groups
+ * small enough to keep their tokens, and the group winners meet in one more prediction.
+ *
* <p>{@code embed_fn_from_agent} is deliberately not ported. It mean-pools the checkpoint's own
* encoder, which needs the encoder's hidden states; the ONNX graph this runtime loads exposes
* logits, not hidden states. A dedicated bi-encoder passed as an {@link Embedder} shortlists
@@ -52,6 +55,9 @@ public final class Shortlist {
/** The reference's default: keep twenty labels. */
public static final int DEFAULT_K = 20;
+ /** The reference's default tournament group: up to 256 labels take one round. */
+ public static final int DEFAULT_TOURNAMENT_GROUP = 16;
+
private Shortlist() {
}
@@ -236,6 +242,178 @@ public final class Shortlist {
return predict(predictor, state, questions, embedder, DEFAULT_K);
}
+ // ------------------------------------------------------------------ the tournament
+
+ /**
+ * What a tournament left of one choice question.
+ *
+ * @param labels the finalists, in criteria order; every label when no round was played
+ * @param total how many labels the question had
+ * @param rounds how many elimination rounds the question played
+ */
+ public record Bracket(List<String> labels, int total, int rounds) {
+
+ public Bracket {
+ labels = List.copyOf(labels);
+ if (labels.isEmpty()) {
+ throw new IllegalArgumentException("a bracket keeps at least one label");
+ }
+ if (total < labels.size()) {
+ throw new IllegalArgumentException(String.format(
+ "total is %d but %d labels were kept", total, labels.size()));
+ }
+ if (rounds < 0) {
+ throw new IllegalArgumentException("rounds must not be negative, got " + rounds);
+ }
+ if (rounds == 0 && labels.size() != total) {
+ throw new IllegalArgumentException(String.format(
+ "a choice that played no round keeps all %d labels, not %d",
+ total, labels.size()));
+ }
+ }
+ }
+
+ /** The final prediction of a tournament, and each choice question's bracket. */
+ public record Tournament(Prediction prediction, Map<String, Bracket> tournament) {
+
+ public Tournament {
+ if (prediction == null) {
+ throw new IllegalArgumentException("prediction must not be null");
+ }
+ if (tournament == null) {
+ throw new IllegalArgumentException("tournament must not be null");
+ }
+ tournament = Collections.unmodifiableMap(new LinkedHashMap<>(tournament));
+ }
+ }
+
+ /**
+ * Narrow each large choice question by elimination, then predict once more.
+ *
+ * <p>A choice with more than {@code groupSize} labels is cut, in criteria order, into groups of
+ * near-equal size and at most {@code groupSize}. One prediction answers every group of every
+ * such question -- the groups go in as separate questions, keyed {@code "0"}, {@code "1"}, ...
+ * -- and each group's answer goes through to the next round. Rounds repeat until no choice has
+ * more than {@code groupSize} labels left.
+ *
+ * <p>The final prediction answers every question of the request, with each choice that played
+ * a round cut to its finalists; everything else goes to it unchanged, so when nothing needs a
+ * round it is the only call. Answers, probabilities and usage are the final call's, so a
+ * tournament choice's probabilities are over its finalists only. The caller's map is not
+ * modified.
+ *
+ * @throws IllegalArgumentException if {@code groupSize} is below two
+ * @throws IllegalStateException if a round's prediction does not answer a group with one of
+ * its labels; the reference fails on such a label at the next
+ * cut instead
+ */
+ public static Tournament predictTournament(Predictor predictor, Object state,
+ Map<String, Question> questions, int groupSize) {
+ if (predictor == null) {
+ throw new IllegalArgumentException("predictor must not be null");
+ }
+ if (questions == null) {
+ throw new IllegalArgumentException("questions must not be null");
+ }
+ if (groupSize < 2) {
+ throw new IllegalArgumentException(
+ "group_size must be an integer of at least 2, got " + groupSize);
+ }
+ Map<String, List<String>> finalists = new LinkedHashMap<>();
+ Map<String, Integer> rounds = new LinkedHashMap<>();
+ for (Map.Entry<String, Question> entry : questions.entrySet()) {
+ Question question = entry.getValue();
+ if (question != null && question.type() == Question.Type.CHOICE) {
+ finalists.put(entry.getKey(), question.labels());
+ rounds.put(entry.getKey(), 0);
+ }
+ }
+ while (true) {
+ List<String> owners = new ArrayList<>();
+ List<List<String>> groups = new ArrayList<>();
+ for (Map.Entry<String, List<String>> entry : finalists.entrySet()) {
+ List<List<String>> parts = groups(entry.getValue(), groupSize);
+ if (parts.size() > 1) {
+ for (List<String> part : parts) {
+ owners.add(entry.getKey());
+ groups.add(part);
+ }
+ }
+ }
+ if (groups.isEmpty()) {
+ break;
+ }
+ Map<String, Question> round = new LinkedHashMap<>();
+ for (int i = 0; i < groups.size(); i++) {
+ round.put(Integer.toString(i), cut(questions.get(owners.get(i)), groups.get(i)));
+ }
+ Prediction answered = predictor.predict(state, round);
+ Map<String, List<String>> winners = new LinkedHashMap<>();
+ for (int i = 0; i < groups.size(); i++) {
+ String id = Integer.toString(i);
+ Answer answer = answered == null ? null : answered.answer(id);
+ if (!(answer instanceof Answer.Choice choice)
+ || !groups.get(i).contains(choice.choice())) {
+ throw new IllegalStateException(String.format(
+ "round question %s of %s was not answered with one of its labels: %s",
+ id, PythonJson.repr(owners.get(i)), answer));
+ }
+ winners.computeIfAbsent(owners.get(i), key -> new ArrayList<>())
+ .add(choice.choice());
+ }
+ for (Map.Entry<String, List<String>> entry : winners.entrySet()) {
+ finalists.put(entry.getKey(), entry.getValue());
+ rounds.merge(entry.getKey(), 1, Integer::sum);
+ }
+ }
+ Map<String, Question> last = new LinkedHashMap<>(questions);
+ Map<String, Bracket> meta = new LinkedHashMap<>();
+ for (Map.Entry<String, List<String>> entry : finalists.entrySet()) {
+ String id = entry.getKey();
+ int played = rounds.get(id);
+ if (played > 0) {
+ last.put(id, cut(questions.get(id), entry.getValue()));
+ }
+ meta.put(id, new Bracket(entry.getValue(), questions.get(id).labels().size(), played));
+ }
+ return new Tournament(predictor.predict(state, last), meta);
+ }
+
+ /** A tournament with the reference's default group size. */
+ public static Tournament predictTournament(Predictor predictor, Object state,
+ Map<String, Question> questions) {
+ return predictTournament(predictor, state, questions, DEFAULT_TOURNAMENT_GROUP);
+ }
+
+ /**
+ * {@code labels} in criteria order, cut into {@code ceil(n / groupSize)} contiguous groups.
+ *
+ * <p>Group {@code i} is {@code labels[i*n/parts : (i+1)*n/parts]}, the reference's integer
+ * split, so sizes differ by at most one and the larger groups fall where it puts them.
+ */
+ static List<List<String>> groups(List<String> labels, int groupSize) {
+ int n = labels.size();
+ int parts = (n + groupSize - 1) / groupSize;
+ List<List<String>> out = new ArrayList<>(parts);
+ for (int i = 0; i < parts; i++) {
+ // long, so i * n cannot overflow on a large label set.
+ int from = (int) ((long) i * n / parts);
+ int to = (int) ((long) (i + 1) * n / parts);
+ out.add(List.copyOf(labels.subList(from, to)));
+ }
+ return out;
+ }
+
+ /** The question asked again over these of its labels, in the order given. */
+ private static Question cut(Question question, List<String> labels) {
+ Map<String, Object> criteria = criteriaOf(question);
+ Map<String, Object> subset = new LinkedHashMap<>();
+ for (String label : labels) {
+ subset.put(label, criteria.get(label));
+ }
+ return Question.choice(question.instructions(), subset);
+ }
+
// ------------------------------------------------------------------ the ranking
/** The one implementation. {@code instructions} may be null, which embeds the state alone. */
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentParityTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentParityTest.java
new file mode 100644
index 00000000..1ad716d3
--- /dev/null
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentParityTest.java
@@ -0,0 +1,150 @@
+package com.convaiinnovations.laya;
+
+import static com.convaiinnovations.laya.PredictionGolden.assertAnswers;
+import static com.convaiinnovations.laya.PredictionGolden.assertUsage;
+import static com.convaiinnovations.laya.PredictionGolden.questionsOf;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.junit.jupiter.api.AfterAll;
+import org.junit.jupiter.api.Assumptions;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.DynamicTest;
+import org.junit.jupiter.api.TestFactory;
+
+/**
+ * {@link Shortlist#predictTournament} on the real graph against {@code tournament_golden.json}.
+ *
+ * <p>Every call is compared: the groups asked (exactly, since the generator refuses a near-tie in
+ * any round), each call's answers within {@link PredictionGolden}'s tolerance, and the usage. Skips
+ * when the golden was recorded without a graph or the checkpoint and graph are not present.
+ */
+final class TournamentParityTest {
+
+ private static Agent shared;
+
+ @AfterAll
+ static void closeAgent() {
+ if (shared != null) {
+ shared.close();
+ shared = null;
+ }
+ }
+
+ @SuppressWarnings("unchecked")
+ private static Map<String, Object> map(Object value) {
+ return (Map<String, Object>) value;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static List<Object> list(Object value) {
+ return (List<Object>) value;
+ }
+
+ @TestFactory
+ @DisplayName("every recorded tournament replays call for call on the real graph")
+ List<DynamicTest> replays() {
+ Map<String, Object> golden = Fixtures.load("tournament_golden.json");
+ Assumptions.assumeFalse(golden.containsKey("skipped"),
+ "tournament_golden.json was recorded without a graph: set " + Fixtures.GRAPH_ENV
+ + " and re-run scripts/gen_fixtures.py");
+ List<DynamicTest> tests = new ArrayList<>();
+ for (Object entry : list(golden.get("cases"))) {
+ Map<String, Object> row = map(entry);
+ String name = (String) row.get("name");
+ tests.add(DynamicTest.dynamicTest(name, () -> {
+ Agent agent = agent(golden);
+ List<Map<String, Question>> asked = new ArrayList<>();
+ List<Prediction> returned = new ArrayList<>();
+ Predictor recording = (state, questions) -> {
+ asked.add(new LinkedHashMap<>(questions));
+ Prediction out = agent.predict(state, questions);
+ returned.add(out);
+ return out;
+ };
+ Object groupSize = row.get("group_size");
+ Map<String, Question> questions = questionsOf(choicesAsMaps(map(row.get("questions"))));
+ Shortlist.Tournament result = groupSize == null
+ ? Shortlist.predictTournament(recording, row.get("state"), questions)
+ : Shortlist.predictTournament(recording, row.get("state"), questions,
+ ((Number) groupSize).intValue());
+
+ List<Object> calls = list(row.get("calls"));
+ assertEquals(calls.size(), asked.size(), name + ": calls");
+ for (int c = 0; c < calls.size(); c++) {
+ Map<String, Object> call = map(calls.get(c));
+ Map<String, Object> expected = map(call.get("asked"));
+ assertEquals(new ArrayList<>(expected.keySet()),
+ new ArrayList<>(asked.get(c).keySet()), name + " call " + c + ": ids");
+ for (Map.Entry<String, Object> q : expected.entrySet()) {
+ Object labels = map(q.getValue()).get("labels");
+ if (labels != null) {
+ assertEquals(labels, asked.get(c).get(q.getKey()).labels(),
+ name + " call " + c + "." + q.getKey() + ": labels");
+ }
+ }
+ assertAnswers(map(call.get("answers")), returned.get(c).answers());
+ assertUsage(map(call.get("usage")), returned.get(c).usage());
+ }
+ assertSame(returned.get(returned.size() - 1), result.prediction(),
+ name + ": answers and usage are the final call's");
+ Map<String, Object> meta = map(row.get("tournament"));
+ assertEquals(new ArrayList<>(meta.keySet()),
+ new ArrayList<>(result.tournament().keySet()));
+ for (Map.Entry<String, Object> bracket : meta.entrySet()) {
+ Map<String, Object> want = map(bracket.getValue());
+ Shortlist.Bracket got = result.tournament().get(bracket.getKey());
+ assertEquals(want.get("labels"), got.labels(), name + ": finalists");
+ assertEquals(((Number) want.get("n")).intValue(), got.total());
+ assertEquals(((Number) want.get("rounds")).intValue(), got.rounds());
+ }
+ }));
+ }
+ assertTrue(tests.size() >= 3, "only " + tests.size() + " recorded tournaments");
+ return tests;
+ }
+
+ /** List criteria on a choice become a map of null descriptions, as the reference reads them. */
+ private static Map<String, Object> choicesAsMaps(Map<String, Object> defs) {
+ Map<String, Object> out = new LinkedHashMap<>();
+ for (Map.Entry<String, Object> entry : defs.entrySet()) {
+ Map<String, Object> q = new LinkedHashMap<>(map(entry.getValue()));
+ if ("choice".equals(q.get("type")) && q.get("criteria") instanceof List) {
+ Map<String, Object> options = new LinkedHashMap<>();
+ for (Object label : list(q.get("criteria"))) {
+ options.put((String) label, null);
+ }
+ q.put("criteria", options);
+ }
+ out.put(entry.getKey(), q);
+ }
+ return out;
+ }
+
+ private static synchronized Agent agent(Map<String, Object> golden) throws IOException {
+ if (shared == null) {
+ String checkpoint = (String) golden.getOrDefault("checkpoint", "multilingual");
+ Path model = Fixtures.checkpoint(checkpoint);
+ Assumptions.assumeTrue(model != null, Fixtures.missingCheckpoint(checkpoint));
+ String graph = System.getenv(Fixtures.GRAPH_ENV);
+ Assumptions.assumeTrue(graph != null && !graph.isBlank(),
+ "set " + Fixtures.GRAPH_ENV + " to the exported laya.onnx this golden was"
+ + " recorded from");
+ Path graphPath = Paths.get(graph);
+ Path directory = Files.isDirectory(graphPath) ? graphPath : graphPath.getParent();
+ Assumptions.assumeTrue(directory != null && Files.isDirectory(directory),
+ Fixtures.GRAPH_ENV + " does not name a graph: " + graph);
+ shared = Agent.open(model, directory);
+ }
+ return shared;
+ }
+}
diff --git a/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentTest.java b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentTest.java
new file mode 100644
index 00000000..e6bb0dcf
--- /dev/null
+++ b/laya-java/laya-java/src/test/java/com/convaiinnovations/laya/TournamentTest.java
@@ -0,0 +1,384 @@
+package com.convaiinnovations.laya;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.DynamicTest;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestFactory;
+
+/**
+ * {@link Shortlist#predictTournament} against the calls recorded from
+ * {@code laya.shortlist.predict_tournament}.
+ *
+ * <p>Both sides answer with the same stand-in model: every choice goes to the label with the
+ * lowest unsigned FNV-1a of {@code rank:<label>}. That makes every round reproducible, so the
+ * fixture pins the groups asked, their order, the winners carried forward and the final cut.
+ */
+class TournamentTest {
+
+ private static final long FNV_OFFSET = 0xCBF29CE484222325L;
+ private static final long FNV_PRIME = 0x100000001B3L;
+
+ private static Map<String, Object> fixture() {
+ return Fixtures.load("tournament.json");
+ }
+
+ @SuppressWarnings("unchecked")
+ private static Map<String, Object> map(Object value) {
+ return (Map<String, Object>) value;
+ }
+
+ @SuppressWarnings("unchecked")
+ private static List<Object> list(Object value) {
+ return (List<Object>) value;
+ }
+
+ private static List<String> strings(Object value) {
+ List<String> out = new ArrayList<>();
+ for (Object item : list(value)) {
+ out.add((String) item);
+ }
+ return out;
+ }
+
+ private static long rank(String label) {
+ long hash = FNV_OFFSET;
+ for (byte value : ("rank:" + label).getBytes(StandardCharsets.UTF_8)) {
+ hash = (hash ^ (value & 0xFFL)) * FNV_PRIME;
+ }
+ return hash;
+ }
+
+ private static Usage noUsage() {
+ return new Usage(0, 0, 0, 0, false, List.of(), Map.of());
+ }
+
+ /** The stand-in model, recording each call: choices only, like the generator's. */
+ private static final class Ranker implements Predictor {
+
+ final List<Map<String, Question>> calls = new ArrayList<>();
+ final List<Prediction> returned = new ArrayList<>();
+ final List<Object> states = new ArrayList<>();
+
+ @Override
+ public Prediction predict(Object state, Map<String, Question> questions) {
+ calls.add(new LinkedHashMap<>(questions));
+ states.add(state);
+ Map<String, Answer> answers = new LinkedHashMap<>();
+ for (Map.Entry<String, Question> entry : questions.entrySet()) {
+ if (entry.getValue().type() != Question.Type.CHOICE) {
+ continue;
+ }
+ String best = null;
+ for (String label : entry.getValue().labels()) {
+ if (best == null || Long.compareUnsigned(rank(label), rank(best)) < 0) {
+ best = label;
+ }
+ }
+ answers.put(entry.getKey(), new Answer.Choice(best, Map.of(best, 1.0), 1, 1, 0));
+ }
+ Prediction out = new Prediction("stub-" + calls.size(), answers, noUsage());
+ returned.add(out);
+ return out;
+ }
+ }
+
+ /** A recorded question; list criteria become a map of null descriptions, as the reference does. */
+ private static Question question(Map<String, Object> spec) {
+ String type = (String) spec.get("type");
+ String instructions = (String) spec.get("instructions");
+ Object criteria = spec.get("criteria");
+ if ("choice".equals(type)) {
+ Map<String, Object> options = new LinkedHashMap<>();
+ if (criteria instanceof List) {
+ for (Object label : list(criteria)) {
+ options.put((String) label, null);
+ }
+ } else {
+ options.putAll(map(criteria));
+ }
+ return Question.choice(instructions, options);
+ }
+ if ("score".equals(type)) {
+ return Question.score(instructions, list(criteria));
+ }
+ return Question.noul(instructions);
+ }
+
+ private static Map<String, Question> questions(Map<String, Object> specs) {
+ Map<String, Question> out = new LinkedHashMap<>();
+ for (Map.Entry<String, Object> entry : specs.entrySet()) {
+ out.put(entry.getKey(), question(map(entry.getValue())));
+ }
+ return out;
+ }
+
+ private static Map<String, Question> oneChoice(int n) {
+ Map<String, Object> criteria = new LinkedHashMap<>();
+ for (int i = 0; i < n; i++) {
+ criteria.put(String.format("l%04d", i), null);
+ }
+ return Map.of("q", Question.choice("pick", criteria));
+ }
+
+ // ------------------------------------------------------------------ golden parity
+
+ @Test
+ @DisplayName("the stand-in model ranks labels as the generator did")
+ void rankMatchesTheGenerator() {
+ Map<String, Object> probe = map(fixture().get("rank_probe"));
+ assertTrue(probe.size() >= 3, "only " + probe.size() + " probe labels");
+ for (Map.Entry<String, Object> entry : probe.entrySet()) {
+ assertEquals(entry.getValue(), Long.toUnsignedString(rank(entry.getKey())),
+ "rank of " + entry.getKey());
+ }
+ assertEquals(((Number) fixture().get("default_group_size")).intValue(),
+ Shortlist.DEFAULT_TOURNAMENT_GROUP);
+ }
+
+ @TestFactory
+ @DisplayName("every recorded split: group sizes per round, finalists and the final choice")
+ List<DynamicTest> splits() {
+ List<DynamicTest> tests = new ArrayList<>();
+ for (Object entry : list(fixture().get("splits"))) {
+ Map<String, Object> row = map(entry);
+ int groupSize = ((Number) row.get("group_size")).intValue();
+ int n = ((Number) row.get("n")).intValue();
+ tests.add(DynamicTest.dynamicTest("n=" + n + " group_size=" + groupSize, () -> {
+ Ranker ranker = new Ranker();
+ Shortlist.Tournament result = Shortlist.predictTournament(ranker, "s",
+ oneChoice(n), groupSize);
+ List<Object> rounds = list(row.get("rounds"));
+ assertEquals(rounds.size() + 1, ranker.calls.size(), "calls");
+ for (int r = 0; r < rounds.size(); r++) {
+ List<Integer> sizes = new ArrayList<>();
+ for (Question asked : ranker.calls.get(r).values()) {
+ sizes.add(asked.labels().size());
+ }
+ List<Integer> expected = new ArrayList<>();
+ for (Object size : list(rounds.get(r))) {
+ expected.add(((Number) size).intValue());
+ }
+ assertEquals(expected, sizes, "group sizes of round " + r);
+ }
+ Shortlist.Bracket bracket = result.tournament().get("q");
+ assertEquals(strings(row.get("finalists")), bracket.labels(), "finalists");
+ assertEquals(rounds.size(), bracket.rounds(), "rounds");
+ assertEquals(n, bracket.total(), "total");
+ assertEquals(row.get("choice"),
+ ((Answer.Choice) result.prediction().answer("q")).choice(), "choice");
+ }));
+ }
+ assertTrue(tests.size() >= 100, "only " + tests.size() + " splits recorded");
+ return tests;
+ }
+
+ @TestFactory
+ @DisplayName("every recorded tournament asks the same calls and reports the same result")
+ List<DynamicTest> cases() {
+ List<DynamicTest> tests = new ArrayList<>();
+ String state = (String) fixture().get("state");
+ for (Object entry : list(fixture().get("cases"))) {
+ Map<String, Object> row = map(entry);
+ String name = (String) row.get("name");
+ tests.add(DynamicTest.dynamicTest(name, () -> {
+ Map<String, Question> questions = questions(map(row.get("questions")));
+ Map<String, Question> original = new LinkedHashMap<>(questions);
+ Ranker ranker = new Ranker();
+ Object groupSize = row.get("group_size");
+ Shortlist.Tournament result = groupSize == null
+ ? Shortlist.predictTournament(ranker, state, questions)
+ : Shortlist.predictTournament(ranker, state, questions,
+ ((Number) groupSize).intValue());
+
+ List<Object> calls = list(row.get("calls"));
+ assertEquals(calls.size(), ranker.calls.size(), name + ": calls");
+ for (int c = 0; c < calls.size(); c++) {
+ assertAsked(name + " call " + c, map(calls.get(c)), ranker.calls.get(c));
+ assertSame(state, ranker.states.get(c), name + ": the state is passed through");
+ }
+ assertEquals(row.get("model"), result.prediction().model(),
+ name + ": the result is the final call's");
+ assertSame(ranker.returned.get(ranker.returned.size() - 1), result.prediction());
+ Map<String, Object> answers = map(row.get("answers"));
+ assertEquals(new ArrayList<>(answers.keySet()),
+ new ArrayList<>(result.prediction().answers().keySet()));
+ for (Map.Entry<String, Object> answer : answers.entrySet()) {
+ assertEquals(answer.getValue(), ((Answer.Choice) result.prediction()
+ .answer(answer.getKey())).choice(), name + "." + answer.getKey());
+ }
+ Map<String, Object> meta = map(row.get("tournament"));
+ assertEquals(new ArrayList<>(meta.keySet()),
+ new ArrayList<>(result.tournament().keySet()), name + ": brackets");
+ for (Map.Entry<String, Object> expected : meta.entrySet()) {
+ Map<String, Object> want = map(expected.getValue());
+ Shortlist.Bracket got = result.tournament().get(expected.getKey());
+ String at = name + "." + expected.getKey();
+ assertEquals(strings(want.get("labels")), got.labels(), at + ".labels");
+ assertEquals(((Number) want.get("n")).intValue(), got.total(), at + ".n");
+ assertEquals(((Number) want.get("rounds")).intValue(), got.rounds(),
+ at + ".rounds");
+ }
+ assertEquals(Boolean.TRUE, row.get("caller_questions_unmutated"));
+ assertEquals(original, questions, name + ": the caller's map was modified");
+ }));
+ }
+ // Floors: a multi-round case and a case where questions play different numbers of rounds.
+ int multiRound = 0;
+ int mixedRounds = 0;
+ for (Object entry : list(fixture().get("cases"))) {
+ List<Integer> played = new ArrayList<>();
+ for (Object bracket : map(map(entry).get("tournament")).values()) {
+ played.add(((Number) map(bracket).get("rounds")).intValue());
+ }
+ multiRound += played.stream().anyMatch(r -> r > 1) ? 1 : 0;
+ mixedRounds += played.stream().distinct().count() > 1 ? 1 : 0;
+ }
+ assertTrue(multiRound >= 1, "no case plays more than one round");
+ assertTrue(mixedRounds >= 1, "no case mixes round counts");
+ return tests;
+ }
+
+ private static void assertAsked(String at, Map<String, Object> expected,
+ Map<String, Question> seen) {
+ assertEquals(new ArrayList<>(expected.keySet()), new ArrayList<>(seen.keySet()),
+ at + ": question ids and order");
+ for (Map.Entry<String, Object> entry : expected.entrySet()) {
+ Map<String, Object> want = map(entry.getValue());
+ Question got = seen.get(entry.getKey());
+ String where = at + "." + entry.getKey();
+ assertEquals(want.get("type"), got.type().wireName(), where + ".type");
+ assertEquals(want.get("instructions"), got.instructions(), where + ".instructions");
+ if (want.containsKey("labels")) {
+ assertEquals(strings(want.get("labels")), got.labels(), where + ".labels");
+ }
+ if (want.containsKey("criteria")) {
+ assertEquals(map(want.get("criteria")), got.spec().get("criteria"),
+ where + ".criteria");
+ }
+ }
+ }
+
+ @TestFactory
+ @DisplayName("a group size below two is refused with the reference's message")
+ List<DynamicTest> refusals() {
+ List<DynamicTest> tests = new ArrayList<>();
+ for (Object entry : list(fixture().get("refusals"))) {
+ Map<String, Object> row = map(entry);
+ int bad = ((Number) row.get("group_size")).intValue();
+ tests.add(DynamicTest.dynamicTest("group_size=" + bad, () -> {
+ assertEquals("ValueError", row.get("error"));
+ IllegalArgumentException refused = assertThrows(IllegalArgumentException.class,
+ () -> Shortlist.predictTournament(new Ranker(), "s", Map.of(), bad));
+ assertEquals(row.get("message"), refused.getMessage());
+ }));
+ }
+ assertTrue(tests.size() >= 3);
+ return tests;
+ }
+
+ // ------------------------------------------------------------------ model-free behaviour
+
+ @Test
+ @DisplayName("77 labels at the default group go out as five balanced groups in criteria order")
+ void balancedGroups() {
+ List<String> labels = new ArrayList<>(oneChoice(77).get("q").labels());
+ List<List<String>> groups = Shortlist.groups(labels, 16);
+ List<Integer> sizes = new ArrayList<>();
+ List<String> joined = new ArrayList<>();
+ for (List<String> group : groups) {
+ sizes.add(group.size());
+ joined.addAll(group);
+ }
+ assertEquals(List.of(15, 15, 16, 15, 16), sizes);
+ assertEquals(labels, joined, "groups are contiguous slices in criteria order");
+ assertEquals(1, Shortlist.groups(labels.subList(0, 16), 16).size());
+ }
+
+ @Test
+ @DisplayName("a consistent ranker's best label wins from any position")
+ void bestLabelWinsFromAnyPosition() {
+ for (int[] shape : new int[][] {{77, 2}, {257, 3}}) {
+ int n = shape[0];
+ List<String> labels = new ArrayList<>(oneChoice(n).get("q").labels());
+ for (String best : List.of(labels.get(0), labels.get(n / 2), labels.get(n - 1))) {
+ List<Map<String, Question>> calls = new ArrayList<>();
+ Predictor favours = (s, qs) -> {
+ calls.add(qs);
+ Map<String, Answer> answers = new LinkedHashMap<>();
+ for (Map.Entry<String, Question> q : qs.entrySet()) {
+ List<String> options = q.getValue().labels();
+ String pick = options.contains(best) ? best : options.get(0);
+ answers.put(q.getKey(), new Answer.Choice(pick, Map.of(), 1, 1, 0));
+ }
+ return new Prediction("stub", answers, noUsage());
+ };
+ Shortlist.Tournament out = Shortlist.predictTournament(favours, "s", oneChoice(n));
+ assertEquals(best, ((Answer.Choice) out.prediction().answer("q")).choice(),
+ n + " labels, best " + best);
+ assertEquals(shape[1], calls.size(), n + " labels, best " + best + ": calls");
+ }
+ }
+ }
+
+ @Test
+ @DisplayName("nothing to narrow: one call, with the caller's questions as given")
+ void nothingToNarrowIsOneCall() {
+ Map<String, Question> asked = new LinkedHashMap<>();
+ asked.put("small", Question.choice("pick", Map.of("x", "")));
+ asked.put("level", Question.score("rate", List.of("low", "high")));
+ Ranker ranker = new Ranker();
+ Shortlist.Tournament out = Shortlist.predictTournament(ranker, "s", asked, 2);
+ assertEquals(1, ranker.calls.size());
+ assertEquals(asked, ranker.calls.get(0));
+ assertSame(asked.get("small"), ranker.calls.get(0).get("small"));
+ assertEquals(0, out.tournament().get("small").rounds());
+ assertEquals(List.of("small"), new ArrayList<>(out.tournament().keySet()));
+ }
+
+ @Test
+ @DisplayName("a round answer outside its group, or missing, is refused")
+ void badRoundAnswerIsRefused() {
+ Predictor outside = (s, qs) -> new Prediction("stub", Map.of("0",
+ new Answer.Choice("l0039", Map.of(), 1, 1, 0)), noUsage());
+ IllegalStateException wrong = assertThrows(IllegalStateException.class,
+ () -> Shortlist.predictTournament(outside, "s", oneChoice(40), 16));
+ assertTrue(wrong.getMessage().contains("round question 0 of 'q'"), wrong.getMessage());
+ Predictor silent = (s, qs) -> new Prediction("stub", Map.of(), noUsage());
+ assertThrows(IllegalStateException.class,
+ () -> Shortlist.predictTournament(silent, "s", oneChoice(40), 16));
+ Predictor nothing = (s, qs) -> null;
+ assertThrows(IllegalStateException.class,
+ () -> Shortlist.predictTournament(nothing, "s", oneChoice(40), 16));
+ assertThrows(IllegalArgumentException.class,
+ () -> Shortlist.predictTournament(nothing, "s", oneChoice(3), 16),
+ "a null final prediction");
+ }
+
+ @Test
+ @DisplayName("null arguments and contradictory brackets are refused")
+ void refusesNullsAndBadBrackets() {
+ assertThrows(IllegalArgumentException.class,
+ () -> Shortlist.predictTournament(null, "s", Map.of()));
+ assertThrows(IllegalArgumentException.class,
+ () -> Shortlist.predictTournament(new Ranker(), "s", null));
+ assertThrows(IllegalArgumentException.class,
+ () -> new Shortlist.Bracket(List.of(), 0, 0));
+ assertThrows(IllegalArgumentException.class,
+ () -> new Shortlist.Bracket(List.of("a", "b"), 1, 1));
+ assertThrows(IllegalArgumentException.class,
+ () -> new Shortlist.Bracket(List.of("a"), 2, 0));
+ assertThrows(IllegalArgumentException.class,
+ () -> new Shortlist.Bracket(List.of("a"), 2, -1));
+ assertEquals(1, new Shortlist.Bracket(List.of("a"), 2, 1).rounds());
+ }
+}
diff --git a/laya-java/scripts/gen_fixtures.py b/laya-java/scripts/gen_fixtures.py
index 0a359c2e..429bc4fa 100644
--- a/laya-java/scripts/gen_fixtures.py
+++ b/laya-java/scripts/gen_fixtures.py
@@ -1104,6 +1104,233 @@ def shortlist():
return payload
+def _tournament_rank(label):
+ """The stand-in model's preference: the lowest FNV-1a of `rank:<label>` wins any group."""
+ return _fnv1a(("rank:%s" % label).encode("utf-8"))
+
+
+class _TournamentRecorder:
+ """Answers every choice with its best-ranked label and records each call, in order."""
+
+ def __init__(self):
+ self.calls = []
+
+ def predict(self, state, questions, **kwargs):
+ from laya import shortlist as mod
+
+ self.calls.append({qid: dict(qdef) for qid, qdef in questions.items()})
+ answers = {}
+ for qid, qdef in questions.items():
+ if qdef.get("type") == "choice":
+ labels = [key for key, _value in mod._criteria_items(qdef["criteria"])]
+ answers[qid] = {"type": "choice", "choice": min(labels, key=_tournament_rank)}
+ return {"model": "stub-%d" % len(self.calls), "answers": answers}
+
+
+def _asked(questions):
+ """A recorded call: each question's type, instructions, labels, and dict criteria verbatim."""
+ from laya import shortlist as mod
+
+ out = {}
+ for qid, qdef in questions.items():
+ entry = {"type": qdef.get("type"), "instructions": qdef.get("instructions")}
+ if qdef.get("type") == "choice":
+ entry["labels"] = [key for key, _value in mod._criteria_items(qdef["criteria"])]
+ if isinstance(qdef["criteria"], dict):
+ entry["criteria"] = qdef["criteria"]
+ out[qid] = entry
+ return out
+
+
+def tournament():
+ """`predict_tournament`'s grouping, rounds and final call, against a deterministic stub model.
+
+ The stub prefers the label with the lowest FNV-1a of `rank:<label>`, reproducible in any
+ language, so every call the reference makes -- the groups of each round, their order, the
+ winners carried forward and the final cut -- can be replayed by a port and compared exactly.
+ """
+ from laya import shortlist as mod
+
+ state = "My card was charged twice for the same order last Tuesday."
+
+ def run(questions, group_size=None):
+ recorder = _TournamentRecorder()
+ before = json.dumps(questions, sort_keys=False)
+ kwargs = {} if group_size is None else {"group_size": group_size}
+ result = mod.predict_tournament(recorder, state, questions, **kwargs)
+ return recorder, result, before == json.dumps(questions, sort_keys=False)
+
+ # Every split size, from the calls the reference actually makes on one bare-label choice.
+ splits = []
+ sweep = [(g, n) for g in (2, 3, 5) for n in range(1, 41)]
+ sweep += [(16, n) for n in (1, 15, 16, 17, 31, 32, 33, 77, 150, 255, 256, 257, 300, 1000)]
+ for group_size, n in sweep:
+ labels = ["l%04d" % i for i in range(n)]
+ recorder, result, _same = run(
+ {"q": {"type": "choice", "instructions": "pick", "criteria": labels}}, group_size)
+ splits.append({
+ "group_size": group_size,
+ "n": n,
+ "rounds": [[len(q["criteria"]) for q in call.values()] for call in recorder.calls[:-1]],
+ "finalists": result["tournament"]["q"]["labels"],
+ "choice": result["answers"]["q"]["choice"],
+ })
+
+ big = {label: "about " + label for label in ["l%02d" % i for i in range(20)]}
+ banking = [
+ "card_arrival", "card_delivery_estimate", "card_not_working", "cash_withdrawal_charge",
+ "declined_card_payment", "direct_debit_payment_not_recognised", "exchange_rate",
+ "failed_transfer", "lost_or_stolen_card", "pending_card_payment",
+ "refund_not_showing_up", "request_refund", "reverted_card_payment", "top_up_failed",
+ "transfer_fee_charged", "verify_my_identity", "wrong_amount_of_cash_received",
+ ]
+ cases = []
+ for name, questions, group_size in [
+ ("default-group-77", {"q": {"type": "choice", "instructions": "pick",
+ "criteria": ["l%03d" % i for i in range(77)]}}, None),
+ ("mixed-group-8", {
+ "big": {"type": "choice", "instructions": "pick", "criteria": big},
+ "small": {"type": "choice", "instructions": "pick", "criteria": ["x", "y"]},
+ "level": {"type": "score", "instructions": "rate", "criteria": ["low", "high"]},
+ "urgent": {"type": "noul", "instructions": "Is this urgent?"},
+ }, 8),
+ ("nothing-to-narrow", {
+ "small": {"type": "choice", "instructions": "pick", "criteria": ["x", "y"]},
+ "level": {"type": "score", "instructions": "rate", "criteria": ["low", "high"]},
+ }, None),
+ ("two-choices-different-rounds", {
+ "wide": {"type": "choice", "instructions": "Which one?",
+ "criteria": ["w%03d" % i for i in range(300)]},
+ "narrow": {"type": "choice", "instructions": "Which other?",
+ "criteria": ["n%02d" % i for i in range(40)]},
+ }, None),
+ ("described-criteria", {"intent": {
+ "type": "choice", "instructions": "What does the customer want?",
+ "criteria": dict([(label, None if i % 4 == 0 else ("" if i % 4 == 1 else
+ (i if i % 4 == 2 else label.replace("_", " "))))
+ for i, label in enumerate(banking)])}}, 3),
+ ("group-of-one", {"q": {"type": "choice", "instructions": "pick",
+ "criteria": ["e", "d", "c", "b", "a"]}}, 2),
+ ("criteria-order-not-sorted", {"q": {"type": "choice", "instructions": "pick",
+ "criteria": list(reversed(banking))}}, 4),
+ ]:
+ recorder, result, unchanged = run(questions, group_size)
+ cases.append({
+ "name": name,
+ "group_size": group_size,
+ "questions": questions,
+ "calls": [_asked(call) for call in recorder.calls],
+ "model": result["model"],
+ "answers": {qid: answer["choice"] for qid, answer in result["answers"].items()},
+ "tournament": result["tournament"],
+ "caller_questions_unmutated": unchanged,
+ })
+
+ refusals = []
+ for bad in (1, 0, -3):
+ try:
+ mod.predict_tournament(_TournamentRecorder(), state, {}, group_size=bad)
+ refusals.append({"group_size": bad, "error": None})
+ except Exception as failure:
+ refusals.append({"group_size": bad, "error": type(failure).__name__,
+ "message": str(failure)})
+
+ return {
+ "default_group_size": mod.DEFAULT_TOURNAMENT_GROUP,
+ "state": state,
+ "rank_probe": {label: str(_tournament_rank(label)) for label in ["", "a", "l0001", "注文"]},
+ "splits": splits,
+ "cases": cases,
+ "refusals": refusals,
+ }
+
+
+TOURNAMENT_GOLDEN_CASES = [
+ ("banking-default-group", "My new card still has not arrived after two weeks.", {
+ "intent": {"type": "choice", "instructions": "What does the customer want?",
+ "criteria": [
+ "activate_my_card", "age_limit", "apple_pay_or_google_pay",
+ "atm_support", "automatic_top_up", "balance_not_updated_after_bank_transfer",
+ "beneficiary_not_allowed", "cancel_transfer", "card_about_to_expire",
+ "card_acceptance", "card_arrival", "card_delivery_estimate",
+ "card_linking", "card_not_working", "card_payment_fee_charged",
+ "card_payment_not_recognised", "card_swallowed", "cash_withdrawal_charge",
+ "change_pin", "compromised_card", "contactless_not_working",
+ "country_support", "declined_card_payment", "declined_cash_withdrawal",
+ "declined_transfer", "edit_personal_details", "exchange_charge",
+ "exchange_rate", "failed_transfer", "fiat_currency_support",
+ "get_physical_card", "lost_or_stolen_card", "lost_or_stolen_phone",
+ "order_physical_card", "passcode_forgotten", "pending_card_payment",
+ "refund_not_showing_up", "request_refund", "terminate_account",
+ "top_up_failed"]},
+ }, None),
+ ("mixed-group-8", "We were billed twice for March and want a refund today.", {
+ "dept": {"type": "choice", "instructions": "Which team owns this?",
+ "criteria": {"billing": "invoices and charges", "refunds": "money sent back",
+ "fraud": "unauthorised activity", "cards": "card problems",
+ "transfers": "bank transfers", "loans": "credit and loans",
+ "savings": "savings products", "kyc": "identity checks",
+ "support": "general help", "legal": "complaints and law",
+ "sales": "new products", "tech": "app bugs"}},
+ "urgent": {"type": "noul", "instructions": "This needs a human today."},
+ }, 8),
+ ("nothing-to-narrow", "The app crashes when I open statements.", {
+ "intent": {"type": "choice", "instructions": "What is wrong?",
+ "criteria": {"bug": "the app fails", "billing": "a charge", "other": "else"}},
+ }, None),
+]
+
+
+def tournament_golden():
+ """`predict_tournament` on a real graph: every call's answers and the final usage.
+
+ Needs an exported graph, like `predict.json`, and records `skipped` without one. A round's
+ winners decide what the next call is asked, so each case is checked here to have a decisive
+ answer in every group: a near-tie would make the replay platform-dependent.
+ """
+ import os
+
+ graph = os.environ.get("LAYA_ONNX_GRAPH")
+ model = os.environ.get("LAYA_PREDICT_MODEL", "multilingual")
+ rig = os.environ.get("LAYA_FIXTURE_CHECKPOINTS", DEFAULT_CHECKPOINT_ROOT)
+ if not graph or not os.path.exists(graph):
+ return {"skipped": "set LAYA_ONNX_GRAPH to an exported laya.onnx to record this family"}
+ checkpoint = os.path.join(rig, model)
+ for needed in ("rl_agent_config.json", "tokenizer/tokenizer.json"):
+ if not os.path.isfile(os.path.join(checkpoint, *needed.split("/"))):
+ return {"skipped": "set LAYA_FIXTURE_CHECKPOINTS to a root whose %s checkpoint has "
+ "%s, to record this family" % (model, needed)}
+ from laya import shortlist as mod
+ from laya.onnx_agent import ONNXAgent
+
+ agent = ONNXAgent(checkpoint, onnx_path=graph)
+
+ class Recording:
+ def __init__(self):
+ self.calls = []
+
+ def predict(self, state, questions, **kwargs):
+ result = agent.predict(state, questions, **kwargs)
+ self.calls.append({"asked": _asked(questions), "answers": result["answers"],
+ "usage": result["usage"]})
+ return result
+
+ cases = []
+ for name, state, questions, group_size in TOURNAMENT_GOLDEN_CASES:
+ recording = Recording()
+ kwargs = {} if group_size is None else {"group_size": group_size}
+ result = mod.predict_tournament(recording, state, questions, **kwargs)
+ for call in recording.calls[:-1]:
+ for qid, answer in call["answers"].items():
+ top = sorted(answer["probabilities"].values(), reverse=True)
+ if len(top) > 1 and top[0] - top[1] <= 1e-3:
+ raise AssertionError("%s: round answer %s is not decisive" % (name, qid))
+ cases.append({"name": name, "state": state, "group_size": group_size,
+ "questions": questions, "calls": recording.calls,
+ "model": result["model"], "tournament": result["tournament"]})
+ return {"checkpoint": model, "graph": os.path.basename(graph), "cases": cases}
+
+
# How many decimals of a cosine are safe to record. The reference computes its scores with
# `np.dot(matrix, vector)`, a BLAS matrix-vector product, and how BLAS blocks that accumulation
# differs between implementations -- measured here: on Apple Accelerate one row of a four-row
@@ -4957,6 +5184,8 @@ FAMILIES = {
"presets.json": presets,
"router.json": router,
"shortlist.json": shortlist,
+ "tournament.json": tournament,
+ "tournament_golden.json": tournament_golden,
"tokenizer_ids.json": tokenizer_ids,
"decode_text.json": decode_text,
"sequences.json": sequences,