Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

feat(ts): add local option stability diagnostics

xpff8d906e8480
diff --git a/laya-ts/README.md b/laya-ts/README.md
index 627e3c93..4451768c 100644
--- a/laya-ts/README.md
+++ b/laya-ts/README.md
@@ -75,6 +75,76 @@ class MetricsHook extends BaseHook {
setDefaultHooks([new MetricsHook()]); // addDefaultHook(...) appends; clearDefaultHooks() resets
```
+### Option stability
+
+`OptionStabilityHook` adds opt-in `reliability` diagnostics to choice answers. It expands
+the questions with reorder and letter-rename probes and evaluates them through the existing
+batch path. The original answer stays in place; internal probe answers and their usage keys
+are removed before the result returns. Extra rows cost inference work even within one call.
+
+```ts
+import { Agent, OptionStabilityHook, calibrateReliability, decideReliability } from "laya-ts";
+
+const agent = await Agent.load("./dist/laya");
+agent.addHook(new OptionStabilityHook({
+ accept: null, escalate: null, variants: "fast", seed: "evaluation-v1",
+}));
+const result = await agent.predict("Please approve the request", {
+ action: { type: "choice", instructions: "Choose action", criteria: ["approve", "reject"] },
+});
+if (result.answers.action.type === "choice") {
+ console.log(result.answers.action.reliability);
+}
+
+// Fit on your labelled examples, using only probed numeric soft_stability values.
+// Keep these scores and cutoffs as numbers; round only a separate display string.
+function fitAndReplay(scores: number[], correct: boolean[], savedScore: number) {
+ const [accept, escalate] = calibrateReliability(scores, correct, { targetAccuracy: 0.9 });
+ return decideReliability(savedScore, accept, escalate);
+}
+```
+
+Both thresholds are required. `accept: null` disables categorical decisions;
+`escalate: null` disables only `ESCALATE`. No calibrated defaults are recommended.
+`ACCEPT` uses `soft_stability >= accept`; `ESCALATE` uses
+`soft_stability < escalate`; otherwise the decision is `VERIFY`. An unprobed question has
+null signals. The report keeps the unrounded mean used by the comparator, matching the
+Python precision correction in #1289. Other diagnostic fractions retain Python's
+four-decimal rounding. Historical rounded Python reports cannot recover the lost precision.
+
+Calibration requires equally sized, nonempty score/boolean-label arrays, finite scores in
+`[0, 1]`, and accuracy targets in `[0, 1]`. Invalid evidence is rejected rather than silently
+dropping unmatched rows. The helpers match Python on valid inputs; they do not copy its
+permissive handling of invalid calibration inputs.
+
+Options use camelCase (`nShuffles`, `renameMaxOptions`); result fields match Python's
+snake_case fields. `variants` accepts `"fast"`, `"full"`, or an array of variant names;
+`probes` accepts `"auto"`, `"reorder"`, `"rename"`, `"all"`, or a probe-name array.
+Renaming retains original label text for missing descriptions, including blank strings,
+as in #1286, and preserves nonblank descriptions and caller `option_order`.
+
+Explicit string seeds and the same ordered labels produce the same permutations as Python's
+CRC32-seeded shuffle. JavaScript enumerates integer-like object keys numerically, so preserve
+the intended label order when comparing inputs across languages.
+They reproduce probe selection, not model outputs. The default seed uses the single string
+state, or `"batch"` for multiple states. A single structured state uses the existing TS state
+serializer; Python uses `repr`, so supply the same explicit seed for cross-language replay.
+The shared fixtures record their Python version and source hashes. Numerical comparison is
+for those decoded inputs; different inference backends or Python floating-point accumulation
+versions can still produce different scores near a threshold.
+
+The hook reads `agent.cfg.option_layout` to choose probes automatically: a declared
+parallel layout selects rename-only probes; sequential selects both. This is probe selection,
+not an implementation of parallel attention. The current local ONNX runtime supports the
+sequential path; this hook adds no new checkpoint/backend support. It also does not add the
+feature to the separate HTTP SDK.
+
+Install the hook once at the Agent, Router, or process-default scope. `predictBatch` and
+concurrent calls can share one instance; plans are scoped to each prediction context and
+released on completion or error. A cache hit without probe answers does not gain a fabricated
+reliability score. The hook's own methods are synchronous and work within the existing async
+Agent/Router prediction lifecycle.
+
## Structured decisions (`decide`)
Turn a JSON schema into typed values in one call — the port of Python's `laya.structured`
diff --git a/laya-ts/fixtures/option-stability.json b/laya-ts/fixtures/option-stability.json
new file mode 100644
index 00000000..aa988871
--- /dev/null
+++ b/laya-ts/fixtures/option-stability.json
@@ -0,0 +1,128 @@
+{
+ "schema": "laya-option-stability/1",
+ "reference": {
+ "base_commit": "ffc142012af63d015fa19f05f29f9c5ff329a2cc",
+ "overlays": [
+ {
+ "commit": "8257e91e4c8c56902720496de75c0a8e2f3f6be7",
+ "pull_request": "https://github.com/NandhaKishorM/laya/pull/1289"
+ },
+ {
+ "commit": "7feac1d3ba4e30ba09bf658f0ec15001b64a9223",
+ "pull_request": "https://github.com/NandhaKishorM/laya/pull/1286"
+ }
+ ],
+ "method": "git show precision commit source; git apply exact rename commit patch for laya/hooks.py; local combined reference, not upstream main",
+ "files": {
+ "laya/hooks.py": "b8f34a77573146b8d6fcdf3d3bc384fac73c52ac612e8fec7218dc3ae9de9321",
+ "laya/common.py": "8396e8c8dbceabfc7fb2abb1e2fe4692751389ba9bd80d592f1ea922fef0765c"
+ },
+ "python": "3.11.13"
+ },
+ "variant_cases": [
+ {"name":"two-dedup","labels":["yes","no"],"seed_key":"","options":{},"expected":[{"name":"original","order":["yes","no"],"relabel":false},{"name":"reversed","order":["no","yes"],"relabel":false},{"name":"letters","order":["yes","no"],"relabel":true},{"name":"letters_reversed","order":["no","yes"],"relabel":true}]},
+ {"name":"three-utf8","labels":["alpha","beta","gamma"],"seed_key":"种子|café|🙂","options":{},"expected":[{"name":"original","order":["alpha","beta","gamma"],"relabel":false},{"name":"reversed","order":["gamma","beta","alpha"],"relabel":false},{"name":"shuffle2","order":["beta","alpha","gamma"],"relabel":false},{"name":"shuffle3","order":["beta","gamma","alpha"],"relabel":false},{"name":"letters","order":["alpha","beta","gamma"],"relabel":true},{"name":"letters_reversed","order":["gamma","beta","alpha"],"relabel":true},{"name":"letters_shuffle","order":["alpha","gamma","beta"],"relabel":true}]},
+ {"name":"twenty-zero-shuffles","labels":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19"],"seed_key":"twenty","options":{"nShuffles":0},"expected":[{"name":"original","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19"],"relabel":false},{"name":"reversed","order":["v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":false},{"name":"letters","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19"],"relabel":true},{"name":"letters_reversed","order":["v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":true},{"name":"letters_shuffle","order":["v14","v5","v1","v11","v8","v6","v17","v12","v9","v7","v0","v3","v4","v15","v10","v16","v2","v13","v18","v19"],"relabel":true}]},
+ {"name":"twenty-six-twenty-shuffles","labels":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25"],"seed_key":"twenty-six","options":{"nShuffles":20},"expected":[{"name":"original","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25"],"relabel":false},{"name":"reversed","order":["v25","v24","v23","v22","v21","v20","v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":false},{"name":"shuffle1","order":["v10","v19","v9","v25","v11","v20","v0","v6","v24","v12","v5","v18","v14","v3","v7","v17","v1","v4","v2","v13","v8","v23","v15","v21","v16","v22"],"relabel":false},{"name":"shuffle2","order":["v14","v8","v19","v0","v21","v18","v16","v22","v4","v6","v5","v2","v3","v7","v17","v24","v20","v11","v13","v9","v1","v10","v12","v25","v15","v23"],"relabel":false},{"name":"shuffle3","order":["v8","v18","v16","v23","v7","v5","v17","v22","v3","v19","v12","v9","v6","v4","v11","v0","v1","v15","v25","v20","v10","v14","v13","v24","v21","v2"],"relabel":false},{"name":"shuffle4","order":["v5","v17","v11","v10","v9","v3","v14","v23","v2","v21","v4","v16","v13","v19","v18","v7","v6","v24","v12","v25","v22","v0","v8","v20","v1","v15"],"relabel":false},{"name":"shuffle5","order":["v16","v6","v22","v12","v25","v2","v5","v1","v24","v7","v19","v17","v9","v10","v21","v20","v8","v15","v23","v14","v4","v11","v18","v13","v0","v3"],"relabel":false},{"name":"shuffle6","order":["v7","v16","v24","v11","v1","v15","v22","v14","v20","v21","v5","v2","v0","v6","v3","v8","v10","v13","v23","v18","v4","v19","v25","v9","v12","v17"],"relabel":false},{"name":"shuffle7","order":["v21","v20","v16","v24","v2","v25","v12","v17","v6","v15","v23","v14","v1","v8","v10","v9","v11","v0","v4","v5","v13","v3","v19","v18","v22","v7"],"relabel":false},{"name":"shuffle8","order":["v25","v4","v12","v20","v19","v16","v10","v6","v9","v14","v1","v18","v24","v3","v22","v0","v5","v15","v21","v8","v2","v23","v11","v13","v7","v17"],"relabel":false},{"name":"shuffle9","order":["v3","v2","v1","v0","v11","v20","v24","v12","v17","v8","v18","v7","v16","v5","v21","v14","v22","v6","v10","v19","v15","v13","v25","v4","v9","v23"],"relabel":false},{"name":"shuffle10","order":["v18","v17","v21","v19","v3","v24","v1","v20","v0","v9","v12","v22","v11","v15","v8","v16","v5","v14","v2","v23","v13","v25","v6","v4","v7","v10"],"relabel":false},{"name":"shuffle11","order":["v2","v24","v8","v19","v17","v15","v13","v16","v5","v11","v23","v9","v10","v14","v25","v21","v1","v12","v22","v7","v4","v20","v3","v0","v18","v6"],"relabel":false},{"name":"shuffle12","order":["v8","v12","v13","v11","v24","v6","v25","v22","v0","v23","v9","v1","v2","v5","v4","v19","v17","v20","v21","v3","v7","v14","v18","v16","v15","v10"],"relabel":false},{"name":"shuffle13","order":["v24","v22","v25","v5","v16","v20","v17","v4","v7","v3","v12","v19","v2","v11","v14","v9","v10","v0","v21","v15","v6","v1","v13","v23","v18","v8"],"relabel":false},{"name":"shuffle14","order":["v17","v20","v21","v22","v16","v11","v0","v10","v14","v25","v4","v13","v8","v9","v15","v19","v1","v7","v5","v24","v23","v6","v3","v12","v2","v18"],"relabel":false},{"name":"shuffle15","order":["v13","v4","v19","v10","v2","v11","v7","v1","v6","v3","v24","v23","v18","v9","v8","v16","v17","v20","v21","v15","v5","v14","v12","v22","v0","v25"],"relabel":false},{"name":"shuffle16","order":["v11","v12","v17","v7","v15","v13","v14","v10","v20","v0","v22","v23","v16","v21","v4","v24","v1","v6","v2","v25","v3","v8","v19","v9","v5","v18"],"relabel":false},{"name":"shuffle17","order":["v2","v8","v4","v3","v22","v14","v20","v12","v5","v7","v17","v23","v13","v15","v21","v16","v10","v0","v24","v18","v19","v1","v11","v6","v25","v9"],"relabel":false},{"name":"shuffle18","order":["v6","v1","v14","v25","v2","v22","v3","v20","v8","v23","v18","v0","v4","v7","v13","v17","v21","v5","v11","v19","v24","v16","v12","v15","v10","v9"],"relabel":false},{"name":"shuffle19","order":["v15","v3","v12","v16","v7","v2","v6","v8","v10","v18","v5","v22","v24","v20","v19","v23","v14","v11","v0","v9","v1","v25","v17","v4","v13","v21"],"relabel":false},{"name":"shuffle20","order":["v7","v17","v1","v5","v16","v25","v11","v12","v15","v18","v2","v4","v6","v23","v14","v19","v24","v13","v22","v10","v21","v3","v8","v20","v9","v0"],"relabel":false},{"name":"letters","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25"],"relabel":true},{"name":"letters_reversed","order":["v25","v24","v23","v22","v21","v20","v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":true},{"name":"letters_shuffle","order":["v1","v3","v22","v8","v2","v11","v5","v4","v15","v14","v20","v21","v12","v16","v24","v18","v13","v10","v9","v7","v25","v23","v19","v6","v17","v0"],"relabel":true}]},
+ {"name":"twenty-seven-no-letters","labels":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25","v26"],"seed_key":"twenty-seven","options":{},"expected":[{"name":"original","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25","v26"],"relabel":false},{"name":"reversed","order":["v26","v25","v24","v23","v22","v21","v20","v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":false},{"name":"shuffle1","order":["v4","v11","v0","v16","v25","v8","v15","v2","v13","v5","v23","v20","v26","v24","v6","v14","v22","v21","v3","v19","v7","v12","v9","v18","v1","v17","v10"],"relabel":false},{"name":"shuffle2","order":["v1","v4","v23","v20","v8","v6","v12","v14","v9","v7","v13","v16","v0","v21","v17","v25","v11","v10","v2","v19","v15","v24","v22","v26","v18","v5","v3"],"relabel":false},{"name":"shuffle3","order":["v5","v6","v0","v8","v24","v17","v19","v22","v4","v23","v18","v11","v15","v20","v12","v2","v16","v9","v10","v1","v13","v21","v26","v7","v14","v3","v25"],"relabel":false}]},
+ {"name":"fast-include","labels":["a","b","c"],"seed_key":"seed","options":{"include":["reversed","letters_reversed"]},"expected":[{"name":"original","order":["a","b","c"],"relabel":false},{"name":"reversed","order":["c","b","a"],"relabel":false},{"name":"letters_reversed","order":["c","b","a"],"relabel":true}]},
+ {"name":"custom-include","labels":["a","b","c","d"],"seed_key":"seed","options":{"include":["shuffle3","letters_shuffle"]},"expected":[{"name":"original","order":["a","b","c","d"],"relabel":false},{"name":"shuffle3","order":["b","d","a","c"],"relabel":false},{"name":"letters_shuffle","order":["a","d","c","b"],"relabel":true}]},
+ {"name":"rename-only","labels":["a","b","c"],"seed_key":"seed","options":{"reorder":false},"expected":[{"name":"original","order":["a","b","c"],"relabel":false},{"name":"letters","order":["a","b","c"],"relabel":true},{"name":"letters_reversed","order":["c","b","a"],"relabel":true}]},
+ {"name":"reorder-only","labels":["a","b","c"],"seed_key":"seed","options":{"relabel":false},"expected":[{"name":"original","order":["a","b","c"],"relabel":false},{"name":"reversed","order":["c","b","a"],"relabel":false},{"name":"shuffle2","order":["a","c","b"],"relabel":false}]},
+ {"name":"no-probes","labels":["a","b","c"],"seed_key":"seed","options":{"relabel":false,"reorder":false},"expected":[{"name":"original","order":["a","b","c"],"relabel":false}]},
+ {"name":"filtered-away","labels":["a","b","c"],"seed_key":"seed","options":{"nShuffles":0,"include":["shuffle3"]},"expected":[{"name":"original","order":["a","b","c"],"relabel":false}]},
+ {"name":"unicode-labels","labels":["","🙂","é"],"seed_key":"nul\u0000seed","options":{"nShuffles":5},"expected":[{"name":"original","order":["","🙂","é"],"relabel":false},{"name":"reversed","order":["é","🙂",""],"relabel":false},{"name":"shuffle1","order":["","é","🙂"],"relabel":false},{"name":"shuffle2","order":["é","","🙂"],"relabel":false},{"name":"shuffle5","order":["🙂","","é"],"relabel":false},{"name":"letters","order":["","🙂","é"],"relabel":true},{"name":"letters_reversed","order":["é","🙂",""],"relabel":true},{"name":"letters_shuffle","order":["🙂","é",""],"relabel":true}]}
+ ],
+ "hook_cases": [
+ {"name":"described-full","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["review: Review the request","accept: Accept the request","reject: Reject the request"]},{"key":"q::rel3","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel4","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]},{"key":"q::rel5","rendered":["A: Review the request","B: Accept the request","C: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}},"q::rel3":{"type":"choice","choice":"A","probabilities":{"A":0.8,"B":0.19999999999999996,"C":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"C":0.25,"B":0.75,"A":0.0}},"q::rel5":{"type":"choice","choice":"B","probabilities":{"B":0.7,"A":0.30000000000000004,"C":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.6667,"reorder_stability":0.5,"n_variants":6,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","shuffle2":"accept","letters":"accept","letters_reversed":"review","letters_shuffle":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"letters":0.8,"letters_reversed":0.25,"letters_shuffle":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"list-fast","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":["accept","review","reject"]}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["accept","review","reject"]},{"key":"q::rel1","rendered":["reject","review","accept"]},{"key":"q::rel2","rendered":["A: reject","B: review","C: accept"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"undescribed","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":null,"review":"","reject":" \t\n"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["accept","review","reject: \t\n"]},{"key":"q::rel1","rendered":["reject: \t\n","review","accept"]},{"key":"q::rel2","rendered":["A: reject","B: review","C: accept"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"mixed-values","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"missing":null,"blank":" ","zero":0,"false":false,"object":{"detail":"café","priority":2},"array":["x",3]}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["missing","blank: ","zero: 0","false: false","object: {\"detail\": \"café\", \"priority\": 2}","array: [\"x\", 3]"]},{"key":"q::rel1","rendered":["array: [\"x\", 3]","object: {\"detail\": \"café\", \"priority\": 2}","false: false","zero: 0","blank: ","missing"]},{"key":"q::rel2","rendered":["A: [\"x\", 3]","B: {\"detail\": \"café\", \"priority\": 2}","C: false","D: 0","E: blank","F: missing"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"missing","probabilities":{"missing":0.8,"blank":0.19999999999999996,"zero":0.0,"false":0.0,"object":0.0,"array":0.0}},"q::rel1":{"type":"choice","choice":"blank","probabilities":{"missing":0.25,"blank":0.75,"zero":0.0,"false":0.0,"object":0.0,"array":0.0}},"q::rel2":{"type":"choice","choice":"F","probabilities":{"F":0.7,"E":0.30000000000000004,"D":0.0,"C":0.0,"B":0.0,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":6,"distinct":6},"q::rel1":{"total":6,"distinct":6},"q::rel2":{"total":6,"distinct":6}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"missing","probabilities":{"missing":0.8,"blank":0.19999999999999996,"zero":0.0,"false":0.0,"object":0.0,"array":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["blank","missing"],"variant_choices":{"original":"missing","reversed":"blank","letters_reversed":"missing"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":6,"distinct":6}}}}]},
+ {"name":"numeric-keys-order","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"1":"first","2":"second","10":"tenth"},"option_order":[2,0,1]}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["10: tenth","1: first","2: second"]},{"key":"q::rel1","rendered":["1: first","10: tenth","2: second"]},{"key":"q::rel2","rendered":["10: tenth","2: second","1: first"]},{"key":"q::rel3","rendered":["C: tenth","A: first","B: second"]},{"key":"q::rel4","rendered":["C: first","A: tenth","B: second"]},{"key":"q::rel5","rendered":["C: tenth","A: second","B: first"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"1","probabilities":{"1":0.8,"2":0.19999999999999996,"10":0.0}},"q::rel1":{"type":"choice","choice":"2","probabilities":{"1":0.25,"2":0.75,"10":0.0}},"q::rel2":{"type":"choice","choice":"1","probabilities":{"1":0.7,"2":0.30000000000000004,"10":0.0}},"q::rel3":{"type":"choice","choice":"A","probabilities":{"A":0.8,"B":0.19999999999999996,"C":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"C":0.25,"B":0.75,"A":0.0}},"q::rel5":{"type":"choice","choice":"B","probabilities":{"B":0.7,"A":0.30000000000000004,"C":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"1","probabilities":{"1":0.8,"2":0.19999999999999996,"10":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.6667,"reorder_stability":0.5,"n_variants":6,"distinct_choices":["1","2"],"variant_choices":{"original":"1","reversed":"2","shuffle2":"1","letters":"1","letters_reversed":"2","letters_shuffle":"1"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"letters":0.8,"letters_reversed":0.25,"letters_shuffle":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"numeric-list","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":["1","2","10"]}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["1","2","10"]},{"key":"q::rel1","rendered":["10","2","1"]},{"key":"q::rel2","rendered":["A: 10","B: 2","C: 1"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"1","probabilities":{"1":0.8,"2":0.19999999999999996,"10":0.0}},"q::rel1":{"type":"choice","choice":"2","probabilities":{"1":0.25,"2":0.75,"10":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"1","probabilities":{"1":0.8,"2":0.19999999999999996,"10":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["1","2"],"variant_choices":{"original":"1","reversed":"2","letters_reversed":"1"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"prototype-labels","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"__proto__":"base","constructor":"create","toString":"print"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["__proto__: base","constructor: create","toString: print"]},{"key":"q::rel1","rendered":["toString: print","constructor: create","__proto__: base"]},{"key":"q::rel2","rendered":["constructor: create","__proto__: base","toString: print"]},{"key":"q::rel3","rendered":["A: base","B: create","C: print"]},{"key":"q::rel4","rendered":["A: print","B: create","C: base"]},{"key":"q::rel5","rendered":["A: create","B: base","C: print"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"__proto__","probabilities":{"__proto__":0.8,"constructor":0.19999999999999996,"toString":0.0}},"q::rel1":{"type":"choice","choice":"constructor","probabilities":{"__proto__":0.25,"constructor":0.75,"toString":0.0}},"q::rel2":{"type":"choice","choice":"__proto__","probabilities":{"__proto__":0.7,"constructor":0.30000000000000004,"toString":0.0}},"q::rel3":{"type":"choice","choice":"A","probabilities":{"A":0.8,"B":0.19999999999999996,"C":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"C":0.25,"B":0.75,"A":0.0}},"q::rel5":{"type":"choice","choice":"B","probabilities":{"B":0.7,"A":0.30000000000000004,"C":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"__proto__","probabilities":{"__proto__":0.8,"constructor":0.19999999999999996,"toString":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.6667,"reorder_stability":0.5,"n_variants":6,"distinct_choices":["__proto__","constructor"],"variant_choices":{"original":"__proto__","reversed":"constructor","shuffle2":"__proto__","letters":"__proto__","letters_reversed":"constructor","letters_shuffle":"__proto__"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"letters":0.8,"letters_reversed":0.25,"letters_shuffle":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"unicode-labels","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"":"private","🙂":"smile","é":"accent"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":[": private","🙂: smile","é: accent"]},{"key":"q::rel1","rendered":["é: accent","🙂: smile",": private"]},{"key":"q::rel2","rendered":["🙂: smile",": private","é: accent"]},{"key":"q::rel3","rendered":["A: private","B: smile","C: accent"]},{"key":"q::rel4","rendered":["A: accent","B: smile","C: private"]},{"key":"q::rel5","rendered":["A: smile","B: private","C: accent"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"","probabilities":{"":0.8,"🙂":0.19999999999999996,"é":0.0}},"q::rel1":{"type":"choice","choice":"🙂","probabilities":{"":0.25,"🙂":0.75,"é":0.0}},"q::rel2":{"type":"choice","choice":"","probabilities":{"":0.7,"🙂":0.30000000000000004,"é":0.0}},"q::rel3":{"type":"choice","choice":"A","probabilities":{"A":0.8,"B":0.19999999999999996,"C":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"C":0.25,"B":0.75,"A":0.0}},"q::rel5":{"type":"choice","choice":"B","probabilities":{"B":0.7,"A":0.30000000000000004,"C":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"","probabilities":{"":0.8,"🙂":0.19999999999999996,"é":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.6667,"reorder_stability":0.5,"n_variants":6,"distinct_choices":["","🙂"],"variant_choices":{"original":"","reversed":"🙂","shuffle2":"","letters":"","letters_reversed":"🙂","letters_shuffle":""},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"letters":0.8,"letters_reversed":0.25,"letters_shuffle":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"custom-variants","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":["shuffle3","letters_shuffle"],"seed":"种子🙂"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["A: Review the request","B: Accept the request","C: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"A","probabilities":{"B":0.25,"A":0.75,"C":0.0}}},"usage":{"input_tokens":200,"output_tokens":2,"truncated_questions":["q","q::rel1"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.525,"stability":0.0,"rename_stability":0.0,"reorder_stability":null,"n_variants":2,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","letters_shuffle":"review"},"variant_support":{"original":0.8,"letters_shuffle":0.25}}}},"usage":{"input_tokens":200,"output_tokens":2,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"zero-shuffles","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"nShuffles":0},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel3","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"A","probabilities":{"A":0.7,"B":0.30000000000000004,"C":0.0}},"q::rel3":{"type":"choice","choice":"C","probabilities":{"C":0.8,"B":0.19999999999999996,"A":0.0}}},"usage":{"input_tokens":400,"output_tokens":4,"truncated_questions":["q","q::rel1","q::rel2","q::rel3"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.6375,"stability":0.6667,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":4,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters":"accept","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters":0.7,"letters_reversed":0.8}}}},"usage":{"input_tokens":400,"output_tokens":4,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"extra-shuffles","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"nShuffles":5},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["review: Review the request","accept: Accept the request","reject: Reject the request"]},{"key":"q::rel3","rendered":["review: Review the request","reject: Reject the request","accept: Accept the request"]},{"key":"q::rel4","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel5","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]},{"key":"q::rel6","rendered":["A: Review the request","B: Accept the request","C: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}},"q::rel3":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"A":0.25,"B":0.75,"C":0.0}},"q::rel5":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}},"q::rel6":{"type":"choice","choice":"B","probabilities":{"B":0.8,"A":0.19999999999999996,"C":0.0}}},"usage":{"input_tokens":700,"output_tokens":7,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5","q::rel6"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3},"q::rel6":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.6142857142857142,"stability":0.6667,"rename_stability":0.6667,"reorder_stability":0.6667,"n_variants":7,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","shuffle2":"accept","shuffle5":"accept","letters":"review","letters_reversed":"accept","letters_shuffle":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"shuffle5":0.8,"letters":0.25,"letters_reversed":0.7,"letters_shuffle":0.8}}}},"usage":{"input_tokens":700,"output_tokens":7,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"uncalibrated","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":null,"escalate":null,"variants":"fast"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":null,"soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"parallel-auto","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel2","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"B","probabilities":{"A":0.25,"B":0.75,"C":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["rename"],"option_layout":"parallel","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":0.5,"reorder_stability":null,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","letters":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"letters":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"parallel-explicit-reorder","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6,"probes":"reorder"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["review: Review the request","accept: Accept the request","reject: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder"],"option_layout":"parallel","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":null,"reorder_stability":0.5,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","shuffle2":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"parallel-relabel-off","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6,"relabel":false},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":[],"option_layout":"parallel","decision":null,"soft_stability":null,"stability":null,"rename_stability":null,"reorder_stability":null,"n_variants":1,"distinct_choices":["accept"],"variant_choices":{"original":"accept"},"variant_support":{"original":0.8}}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"rename-cap","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"v0":"description 0","v1":"description 1","v2":"description 2","v3":"description 3","v4":"description 4","v5":"description 5","v6":"description 6","v7":"description 7","v8":"description 8","v9":"description 9","v10":"description 10","v11":"description 11","v12":"description 12","v13":"description 13","v14":"description 14","v15":"description 15","v16":"description 16","v17":"description 17","v18":"description 18","v19":"description 19"}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["v0: description 0","v1: description 1","v2: description 2","v3: description 3","v4: description 4","v5: description 5","v6: description 6","v7: description 7","v8: description 8","v9: description 9","v10: description 10","v11: description 11","v12: description 12","v13: description 13","v14: description 14","v15: description 15","v16: description 16","v17: description 17","v18: description 18","v19: description 19"]},{"key":"q::rel1","rendered":["A: description 19","B: description 18","C: description 17","D: description 16","E: description 15","F: description 14","G: description 13","H: description 12","I: description 11","J: description 10","K: description 9","L: description 8","M: description 7","N: description 6","O: description 5","P: description 4","Q: description 3","R: description 2","S: description 1","T: description 0"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0}},"q::rel1":{"type":"choice","choice":"S","probabilities":{"T":0.25,"S":0.75,"R":0.0,"Q":0.0,"P":0.0,"O":0.0,"N":0.0,"M":0.0,"L":0.0,"K":0.0,"J":0.0,"I":0.0,"H":0.0,"G":0.0,"F":0.0,"E":0.0,"D":0.0,"C":0.0,"B":0.0,"A":0.0}}},"usage":{"input_tokens":200,"output_tokens":2,"truncated_questions":["q","q::rel1"],"options":{"q":{"total":20,"distinct":20},"q::rel1":{"total":20,"distinct":20}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0},"reliability":{"probes":["rename"],"option_layout":"parallel","decision":"ESCALATE","soft_stability":0.525,"stability":0.0,"rename_stability":0.0,"reorder_stability":null,"n_variants":2,"distinct_choices":["v0","v1"],"variant_choices":{"original":"v0","letters_reversed":"v1"},"variant_support":{"original":0.8,"letters_reversed":0.25}}}},"usage":{"input_tokens":200,"output_tokens":2,"truncated_questions":["q"],"options":{"q":{"total":20,"distinct":20}}}}]},
+ {"name":"over-rename-cap","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"v0":null,"v1":null,"v2":null,"v3":null,"v4":null,"v5":null,"v6":null,"v7":null,"v8":null,"v9":null,"v10":null,"v11":null,"v12":null,"v13":null,"v14":null,"v15":null,"v16":null,"v17":null,"v18":null,"v19":null,"v20":null}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0,"v20":0.0}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":21,"distinct":21}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0,"v20":0.0},"reliability":{"probes":[],"option_layout":"parallel","decision":null,"soft_stability":null,"stability":null,"rename_stability":null,"reorder_stability":null,"n_variants":1,"distinct_choices":["v0"],"variant_choices":{"original":"v0"},"variant_support":{"original":0.8}}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":21,"distinct":21}}}}]},
+ {"name":"alphabet-cap","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"v0":null,"v1":null,"v2":null,"v3":null,"v4":null,"v5":null,"v6":null,"v7":null,"v8":null,"v9":null,"v10":null,"v11":null,"v12":null,"v13":null,"v14":null,"v15":null,"v16":null,"v17":null,"v18":null,"v19":null,"v20":null,"v21":null,"v22":null,"v23":null,"v24":null,"v25":null,"v26":null}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6,"renameMaxOptions":27},"expanded":[{"key":"q","rendered":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25","v26"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0,"v20":0.0,"v21":0.0,"v22":0.0,"v23":0.0,"v24":0.0,"v25":0.0,"v26":0.0}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":27,"distinct":27}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0,"v20":0.0,"v21":0.0,"v22":0.0,"v23":0.0,"v24":0.0,"v25":0.0,"v26":0.0},"reliability":{"probes":[],"option_layout":"parallel","decision":null,"soft_stability":null,"stability":null,"rename_stability":null,"reorder_stability":null,"n_variants":1,"distinct_choices":["v0"],"variant_choices":{"original":"v0"},"variant_support":{"original":0.8}}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":27,"distinct":27}}}}]},
+ {"name":"one-option","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"only":null}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["only"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"only","probabilities":{"only":1.0}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":1,"distinct":1}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"only","probabilities":{"only":1.0},"reliability":{"probes":[],"option_layout":"sequential","decision":null,"soft_stability":null,"stability":null,"rename_stability":null,"reorder_stability":null,"n_variants":1,"distinct_choices":["only"],"variant_choices":{"original":"only"},"variant_support":{"original":1.0}}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":1,"distinct":1}}}}]},
+ {"name":"batch-seed","states":["first","second"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}},{"answers":{"q":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel1":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.8,"B":0.19999999999999996,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}},{"answers":{"q":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.4166666666666667,"stability":0.0,"rename_stability":0.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"review","reversed":"accept","letters_reversed":"accept"},"variant_support":{"original":0.75,"reversed":0.3,"letters_reversed":0.2}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"explicit-structured-state","states":[{"user":"你好","enabled":true}],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"seed":"shared seed"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["reject: Reject the request","accept: Accept the request","review: Review the request"]},{"key":"q::rel3","rendered":["accept: Accept the request","reject: Reject the request","review: Review the request"]},{"key":"q::rel4","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel5","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}},"q::rel3":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"A":0.25,"B":0.75,"C":0.0}},"q::rel5":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.5,"reorder_stability":0.6667,"n_variants":6,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","shuffle1":"accept","shuffle2":"accept","letters":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle1":0.7,"shuffle2":0.8,"letters":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},
+ {"name":"rounding-exact-halves","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.9,"no":0.09999999999999998}},"q::rel1":{"type":"choice","choice":"no","probabilities":{"yes":0.03125,"no":0.96875}},"q::rel2":{"type":"choice","choice":"A","probabilities":{"B":0.15625,"A":0.84375}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.9,"no":0.09999999999999998},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.3625,"stability":0.0,"rename_stability":0.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["no","yes"],"variant_choices":{"original":"yes","reversed":"no","letters_reversed":"no"},"variant_support":{"original":0.9,"reversed":0.0312,"letters_reversed":0.1562}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},
+ {"name":"rounding-binary-neighbors","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.9,"no":0.09999999999999998}},"q::rel1":{"type":"choice","choice":"no","probabilities":{"yes":0.12345,"no":0.8765499999999999}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.99995,"A":4.999999999999449e-05}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.9,"no":0.09999999999999998},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.6744666666666667,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["no","yes"],"variant_choices":{"original":"yes","reversed":"no","letters_reversed":"yes"},"variant_support":{"original":0.9,"reversed":0.1235,"letters_reversed":1.0}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},
+ {"name":"below-accept","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.7999,"no":0.20009999999999994}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.8,"A":0.19999999999999996}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.7999,"no":0.20009999999999994},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.7999666666666667,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.7999,"reversed":0.8,"letters_reversed":0.8}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},
+ {"name":"at-accept","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.8,"A":0.19999999999999996}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ACCEPT","soft_stability":0.8000000000000002,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.8,"reversed":0.8,"letters_reversed":0.8}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},
+ {"name":"above-accept","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.8001,"no":0.19989999999999997}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.8,"A":0.19999999999999996}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.8001,"no":0.19989999999999997},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ACCEPT","soft_stability":0.8000333333333334,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.8001,"reversed":0.8,"letters_reversed":0.8}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},
+ {"name":"below-escalate","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.5999,"no":0.4001}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.6,"no":0.4}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.6,"A":0.4}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.5999,"no":0.4001},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5999666666666666,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.5999,"reversed":0.6,"letters_reversed":0.6}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},
+ {"name":"at-escalate","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.6,"no":0.4}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.6,"no":0.4}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.6,"A":0.4}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.6,"no":0.4},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.6,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.6,"reversed":0.6,"letters_reversed":0.6}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]}
+ ],
+ "calibration_cases": [
+ {"name":"near-equal-cuts","scores":[0.79996,0.8,0.80004],"correct":[false,false,true],"options":{"targetAccuracy":1.0},"thresholds":[0.80004,0.8],"decisions":["ESCALATE","VERIFY","ACCEPT"]},
+ {"name":"mixed-neighbors","scores":[0.59996,0.6,0.60004,0.79996,0.8,0.80004],"correct":[false,true,false,true,true,true],"options":{},"thresholds":[0.79996,0.6],"decisions":["ESCALATE","VERIFY","VERIFY","ACCEPT","ACCEPT","ACCEPT"]},
+ {"name":"equal-score-cohort","scores":[0.8,0.8,0.9],"correct":[true,false,true],"options":{"targetAccuracy":1.0},"thresholds":[0.9,0.0],"decisions":["VERIFY","VERIFY","ACCEPT"]},
+ {"name":"accept-nothing","scores":[0.2,0.8,0.9],"correct":[false,false,false],"options":{},"thresholds":[1.01,0.9],"decisions":["ESCALATE","ESCALATE","VERIFY"]},
+ {"name":"all-correct","scores":[0.2,0.8,0.9],"correct":[true,true,true],"options":{},"thresholds":[0.2,0.0],"decisions":["ACCEPT","ACCEPT","ACCEPT"]},
+ {"name":"edges","scores":[0.0,1.0],"correct":[false,true],"options":{"maxEscalateAccuracy":0.0},"thresholds":[1.0,0.0],"decisions":["VERIFY","ACCEPT"]}
+ ],
+ "decision_cases": [
+ {"score":0.0,"accept":null,"escalate":null,"expected":null},
+ {"score":0.5999999999999999,"accept":null,"escalate":null,"expected":null},
+ {"score":0.6,"accept":null,"escalate":null,"expected":null},
+ {"score":0.7999999999999999,"accept":null,"escalate":null,"expected":null},
+ {"score":0.8,"accept":null,"escalate":null,"expected":null},
+ {"score":0.8000000000000002,"accept":null,"escalate":null,"expected":null},
+ {"score":1.0,"accept":null,"escalate":null,"expected":null},
+ {"score":0.0,"accept":0.8,"escalate":0.6,"expected":"ESCALATE"},
+ {"score":0.5999999999999999,"accept":0.8,"escalate":0.6,"expected":"ESCALATE"},
+ {"score":0.6,"accept":0.8,"escalate":0.6,"expected":"VERIFY"},
+ {"score":0.7999999999999999,"accept":0.8,"escalate":0.6,"expected":"VERIFY"},
+ {"score":0.8,"accept":0.8,"escalate":0.6,"expected":"ACCEPT"},
+ {"score":0.8000000000000002,"accept":0.8,"escalate":0.6,"expected":"ACCEPT"},
+ {"score":1.0,"accept":0.8,"escalate":0.6,"expected":"ACCEPT"},
+ {"score":0.0,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},
+ {"score":0.5999999999999999,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},
+ {"score":0.6,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},
+ {"score":0.7999999999999999,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},
+ {"score":0.8,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},
+ {"score":0.8000000000000002,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},
+ {"score":1.0,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},
+ {"score":0.0,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},
+ {"score":0.5999999999999999,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},
+ {"score":0.6,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},
+ {"score":0.7999999999999999,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},
+ {"score":0.8,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},
+ {"score":0.8000000000000002,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},
+ {"score":1.0,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"}
+ ],
+ "pipeline": {
+ "cases": [
+ "below-accept",
+ "at-accept",
+ "above-accept"
+ ],
+ "correct": [
+ false,
+ false,
+ true
+ ],
+ "options": {
+ "targetAccuracy": 1.0
+ },
+ "thresholds": [
+ 0.8000333333333334,
+ 0.8000000000000002
+ ],
+ "decisions": [
+ "ESCALATE",
+ "VERIFY",
+ "ACCEPT"
+ ]
+ }
+}
diff --git a/laya-ts/scripts/option_stability_fixtures.py b/laya-ts/scripts/option_stability_fixtures.py
new file mode 100644
index 00000000..004c2169
--- /dev/null
+++ b/laya-ts/scripts/option_stability_fixtures.py
@@ -0,0 +1,276 @@
+"""Generate model-free TypeScript fixtures from production Python stability hooks.
+
+ python laya-ts/scripts/option_stability_fixtures.py --python-root REFERENCE
+ python laya-ts/scripts/option_stability_fixtures.py --python-root REFERENCE --check
+
+REFERENCE must contain laya/hooks.py and laya/common.py. The committed oracle records
+its actual source hashes and Git provenance; it uses the rename semantics from #1286
+and the raw score/report/calibration contract from #1289. It is a local combination
+of those changes until they land, not a claim that they are already on main.
+Only the standard library is needed. The production render functions are loaded
+unchanged from their AST so generating fixtures does not import torch or a model.
+"""
+from __future__ import annotations
+
+import argparse
+import ast
+from copy import deepcopy
+import hashlib
+import importlib.util
+import json
+import math
+from pathlib import Path
+import platform
+import subprocess
+import sys
+from types import SimpleNamespace
+
+
+ROOT = Path(__file__).resolve().parents[2]
+FIXTURE = ROOT / "laya-ts/fixtures/option-stability.json"
+OPTION_NAMES = {
+ "nShuffles": "n_shuffles", "renameMaxOptions": "rename_max_options",
+ "targetAccuracy": "target_accuracy", "maxEscalateAccuracy": "max_escalate_accuracy",
+}
+
+
+def python_options(options):
+ return {OPTION_NAMES.get(key, key): value for key, value in options.items()}
+
+
+def load_reference(root):
+ path = root / "laya/hooks.py"
+ spec = importlib.util.spec_from_file_location("stability_fixture_reference", path)
+ hooks = importlib.util.module_from_spec(spec)
+ sys.modules[spec.name] = hooks
+ spec.loader.exec_module(hooks)
+ render_names = {"render_criterion", "_resolve_noul_labels", "render_options"}
+ tree = ast.parse((root / "laya/common.py").read_text(encoding="utf-8"))
+ functions = [node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name in render_names]
+ if {node.name for node in functions} != render_names:
+ raise RuntimeError("reference common.py is missing the production render helpers")
+ scope = {"json": json, "Dict": dict, "List": list,
+ "_DEFAULT_NOUL_LABELS": {"false": "false", "true": "true"}}
+ exec(compile(ast.Module(body=functions, type_ignores=[]), str(root / "laya/common.py"), "exec"), scope)
+ return hooks, scope["render_options"]
+
+
+def provenance(root):
+ hashes = {name: hashlib.sha256((root / name).read_bytes()).hexdigest()
+ for name in ("laya/hooks.py", "laya/common.py")}
+ manifest = root / "reference-provenance.json"
+ if manifest.exists():
+ source = json.loads(manifest.read_text(encoding="utf-8"))
+ if source.get("files") != hashes:
+ raise RuntimeError("reference files differ from their provenance manifest")
+ else:
+ def git(*args):
+ return subprocess.check_output(["git", "-C", str(root), *args], text=True).strip()
+ source = {"commit": git("rev-parse", "HEAD"),
+ "modified_reference_files": git("status", "--porcelain", "--", "laya/hooks.py", "laya/common.py"),
+ "files": hashes}
+ return {**source, "python": platform.python_version()}
+
+
+def variants(hooks):
+ cases = [
+ ("two-dedup", ["yes", "no"], "", {}),
+ ("three-utf8", ["alpha", "beta", "gamma"], "种子|café|🙂", {}),
+ ("twenty-zero-shuffles", [f"v{i}" for i in range(20)], "twenty", {"nShuffles": 0}),
+ # Enough draws to cross MT19937's 624-word state boundary, not only its first block.
+ ("twenty-six-twenty-shuffles", [f"v{i}" for i in range(26)], "twenty-six", {"nShuffles": 20}),
+ ("twenty-seven-no-letters", [f"v{i}" for i in range(27)], "twenty-seven", {}),
+ ("fast-include", ["a", "b", "c"], "seed", {"include": ["reversed", "letters_reversed"]}),
+ ("custom-include", ["a", "b", "c", "d"], "seed", {"include": ["shuffle3", "letters_shuffle"]}),
+ ("rename-only", ["a", "b", "c"], "seed", {"reorder": False}),
+ ("reorder-only", ["a", "b", "c"], "seed", {"relabel": False}),
+ ("no-probes", ["a", "b", "c"], "seed", {"relabel": False, "reorder": False}),
+ ("filtered-away", ["a", "b", "c"], "seed", {"nShuffles": 0, "include": ["shuffle3"]}),
+ ("unicode-labels", ["\ue000", "🙂", "é"], "nul\u0000seed", {"nShuffles": 5}),
+ ]
+ return [{"name": name, "labels": labels, "seed_key": seed, "options": options,
+ "expected": [{"name": n, "order": order, "relabel": rel}
+ for n, order, rel in hooks.make_variants(labels, seed, **python_options(options))]}
+ for name, labels, seed, options in cases]
+
+
+def hook_case(hooks, render_options, name, criteria, *, options=None, layout="sequential",
+ order=None, states=None, supports=None):
+ options = {"accept": 0.8, "escalate": 0.6, **(options or {})}
+ q = {"type": "choice", "instructions": "Choose a suitable action.", "criteria": criteria}
+ if order is not None:
+ q["option_order"] = order
+ questions = {"q": q}
+ original = deepcopy(questions)
+ states = ["fixture state"] if states is None else states
+ hook = hooks.OptionStabilityHook(**python_options(options))
+ ctx = hooks.PredictContext(states=deepcopy(states), questions=questions,
+ agent=SimpleNamespace(config={"option_layout": layout}))
+ hook.on_predict_start(ctx)
+ assert questions == original, "the reference mutated caller-owned questions"
+
+ expanded = []
+ for key, question in ctx.questions.items():
+ crit = question["criteria"]
+ canonical = dict.fromkeys(crit) if isinstance(crit, list) else crit
+ rendered = render_options({"t": "choice", "crit": canonical})
+ physical = question.get("option_order", list(range(len(rendered))))
+ expanded.append({"key": key, "rendered": [rendered[i] for i in physical]})
+
+ # The reference's map is only used to synthesize model rows in each probe's
+ # displayed label space. Expected public results still come from on_predict_end.
+ plan = hook._plans[ctx.run_id]["q"]["variants"]
+ labels = list(criteria)
+ target = labels[0]
+ others = labels[1:]
+ model_results = []
+ for state_index in range(len(states)):
+ answers = {}
+ for i, (_, key, back, _) in enumerate(plan):
+ support = supports[i % len(supports)] if supports else (0.8, 0.25, 0.7)[(i + state_index) % 3]
+ if not others:
+ support = 1.0
+ chosen = target if support >= 0.5 or not others else others[0]
+ forward = {original: shown for shown, original in back.items()}
+ probabilities = {forward[label]: (support if label == target else
+ 1.0 - support if label == others[0] else 0.0) for label in labels}
+ answers[key] = {"type": "choice", "choice": forward[chosen], "probabilities": probabilities}
+ usage = {"input_tokens": 100 * len(answers), "output_tokens": len(answers),
+ "truncated_questions": list(answers),
+ "options": {key: {"total": len(labels), "distinct": len(labels)} for key in answers}}
+ model_results.append({"answers": answers, "usage": usage})
+ ctx.results = deepcopy(model_results)
+ hook.on_predict_end(ctx)
+ assert questions == original, "end hook mutated caller-owned questions"
+ assert ctx.run_id not in hook._plans, "reference leaked its completed plan"
+ for result in ctx.results:
+ report = result["answers"]["q"]["reliability"]
+ if report["soft_stability"] is not None:
+ assert hooks.decide_reliability(report["soft_stability"], options["accept"], options["escalate"]) == report["decision"]
+ return {"name": name, "states": states, "questions": original, "layout": layout,
+ "options": options, "expanded": expanded,
+ "model_results": model_results, "expected_results": ctx.results}
+
+
+def hook_cases(hooks, render_options):
+ make = lambda name, criteria, **kw: hook_case(hooks, render_options, name, criteria, **kw)
+ described = {"accept": "Accept the request", "review": "Review the request", "reject": "Reject the request"}
+ cases = [
+ make("described-full", described),
+ make("list-fast", list(described), options={"variants": "fast"}),
+ make("undescribed", {"accept": None, "review": "", "reject": " \t\n"}, options={"variants": "fast"}),
+ make("mixed-values", {"missing": None, "blank": " ", "zero": 0, "false": False,
+ "object": {"detail": "café", "priority": 2}, "array": ["x", 3]},
+ options={"variants": "fast"}),
+ make("numeric-keys-order", {"1": "first", "2": "second", "10": "tenth"}, order=[2, 0, 1]),
+ make("numeric-list", ["1", "2", "10"], options={"variants": "fast"}),
+ make("prototype-labels", {"__proto__": "base", "constructor": "create", "toString": "print"}),
+ make("unicode-labels", {"\ue000": "private", "🙂": "smile", "é": "accent"}),
+ make("custom-variants", described, options={"variants": ["shuffle3", "letters_shuffle"], "seed": "种子🙂"}),
+ make("zero-shuffles", described, options={"nShuffles": 0}),
+ make("extra-shuffles", described, options={"nShuffles": 5}),
+ make("uncalibrated", described, options={"accept": None, "escalate": None, "variants": "fast"}),
+ make("parallel-auto", described, layout="parallel"),
+ make("parallel-explicit-reorder", described, layout="parallel", options={"probes": "reorder"}),
+ make("parallel-relabel-off", described, layout="parallel", options={"relabel": False}),
+ make("rename-cap", {f"v{i}": f"description {i}" for i in range(20)}, layout="parallel", options={"variants": "fast"}),
+ make("over-rename-cap", {f"v{i}": None for i in range(21)}, layout="parallel"),
+ make("alphabet-cap", {f"v{i}": None for i in range(27)}, layout="parallel", options={"renameMaxOptions": 27}),
+ make("one-option", {"only": None}),
+ make("batch-seed", described, states=["first", "second"], options={"variants": "fast"}),
+ make("explicit-structured-state", described, states=[{"user": "你好", "enabled": True}], options={"seed": "shared seed"}),
+ make("rounding-exact-halves", {"yes": "allow", "no": "deny"},
+ options={"variants": "fast"}, supports=[0.9, 0.03125, 0.15625]),
+ make("rounding-binary-neighbors", {"yes": "allow", "no": "deny"},
+ options={"variants": "fast"}, supports=[0.9, 0.12345, 0.99995]),
+ ]
+ for name, support in [("below-accept", [0.7999, 0.8, 0.8]),
+ ("at-accept", [0.8, 0.8, 0.8]),
+ ("above-accept", [0.8001, 0.8, 0.8]),
+ ("below-escalate", [0.5999, 0.6, 0.6]),
+ ("at-escalate", [0.6, 0.6, 0.6])]:
+ case = make(name, {"yes": "allow", "no": "deny"}, options={"variants": "fast"}, supports=support)
+ report = case["expected_results"][0]["answers"]["q"]["reliability"]
+ assert report["soft_stability"] == sum(support) / 3, "reference needs #1289 raw scores"
+ cases.append(case)
+ renamed = next(case for case in cases if case["name"] == "undescribed")["expanded"][-1]["rendered"]
+ assert renamed == ["A: reject", "B: review", "C: accept"], "reference needs #1286 label semantics"
+ return cases
+
+
+def calibrations(hooks):
+ cases = [
+ ("near-equal-cuts", [0.79996, 0.8, 0.80004], [False, False, True], {"targetAccuracy": 1.0}),
+ ("mixed-neighbors", [0.59996, 0.6, 0.60004, 0.79996, 0.8, 0.80004], [False, True, False, True, True, True], {}),
+ ("equal-score-cohort", [0.8, 0.8, 0.9], [True, False, True], {"targetAccuracy": 1.0}),
+ ("accept-nothing", [0.2, 0.8, 0.9], [False, False, False], {}),
+ ("all-correct", [0.2, 0.8, 0.9], [True, True, True], {}),
+ ("edges", [0.0, 1.0], [False, True], {"maxEscalateAccuracy": 0.0}),
+ ]
+ result = []
+ for name, scores, correct, options in cases:
+ thresholds = hooks.calibrate_reliability(scores, correct, **python_options(options))
+ result.append({"name": name, "scores": scores, "correct": correct, "options": options,
+ "thresholds": thresholds,
+ "decisions": [hooks.decide_reliability(score, *thresholds) for score in scores]})
+ assert result[0]["thresholds"][0] == 0.80004, "reference needs #1289 distinct raw cuts"
+ return result
+
+
+def generate(root):
+ hooks, render_options = load_reference(root)
+ cases = hook_cases(hooks, render_options)
+ decisions = [{"score": score, "accept": accept, "escalate": escalate,
+ "expected": hooks.decide_reliability(score, accept, escalate)}
+ for accept, escalate in [(None, None), (0.8, 0.6), (1.01, 0.0), (0.0, 0.0)]
+ for score in [0.0, math.nextafter(0.6, 0.0), 0.6, math.nextafter(0.8, 0.0), 0.8,
+ math.nextafter(0.8, 1.0), 1.0]]
+ selected = [case for case in cases if case["name"] in ("below-accept", "at-accept", "above-accept")]
+ scores = [case["expected_results"][0]["answers"]["q"]["reliability"]["soft_stability"] for case in selected]
+ correct = [False, False, True]
+ thresholds = hooks.calibrate_reliability(scores, correct, target_accuracy=1.0)
+ return {"schema": "laya-option-stability/1", "reference": provenance(root),
+ "variant_cases": variants(hooks), "hook_cases": cases,
+ "calibration_cases": calibrations(hooks), "decision_cases": decisions,
+ "pipeline": {"cases": [case["name"] for case in selected], "correct": correct,
+ "options": {"targetAccuracy": 1.0}, "thresholds": thresholds,
+ "decisions": [hooks.decide_reliability(score, *thresholds) for score in scores]}}
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--python-root", type=Path, default=ROOT)
+ parser.add_argument("--fixture", type=Path, default=FIXTURE)
+ parser.add_argument("--check", action="store_true", help="compare behavior with the committed oracle without writing")
+ args = parser.parse_args()
+ result = generate(args.python_root.resolve())
+ if args.check:
+ expected = json.loads(args.fixture.read_text(encoding="utf-8"))
+ # A newer source commit may implement the same contract. Compare behavior;
+ # print actual provenance, retaining the original oracle's provenance.
+ actual = {key: value for key, value in result.items() if key != "reference"}
+ wanted = {key: value for key, value in expected.items() if key != "reference"}
+ if json.loads(json.dumps(actual)) != wanted:
+ raise SystemExit("option stability fixtures differ from this Python reference")
+ print("PASS: Python reference matches all committed option stability fixtures")
+ else:
+ args.fixture.parent.mkdir(parents=True, exist_ok=True)
+ # Keep one complete case per line: the oracle stays small while its named
+ # cases and source provenance remain easy to locate and inspect.
+ fields = []
+ for key, value in result.items():
+ if key.endswith("_cases"):
+ encoded = "[\n" + ",\n".join(" " + json.dumps(case, ensure_ascii=False, separators=(",", ":"))
+ for case in value) + "\n ]"
+ else:
+ encoded = json.dumps(value, ensure_ascii=False, indent=2).replace("\n", "\n ")
+ fields.append(f" {json.dumps(key)}: {encoded}")
+ args.fixture.write_text("{\n" + ",\n".join(fields) + "\n}\n", encoding="utf-8")
+ print(f"Wrote {args.fixture} ({args.fixture.stat().st_size} bytes)")
+ print(json.dumps(result["reference"], ensure_ascii=False, sort_keys=True))
+ print(f"Cases: {len(result['variant_cases'])} variants, {len(result['hook_cases'])} hook calls, "
+ f"{len(result['calibration_cases'])} calibrations, {len(result['decision_cases'])} decisions, 1 fit/replay pipeline")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/laya-ts/src/_option-stability-random.ts b/laya-ts/src/_option-stability-random.ts
new file mode 100644
index 00000000..c87a7ea6
--- /dev/null
+++ b/laya-ts/src/_option-stability-random.ts
@@ -0,0 +1,65 @@
+/** Python's integer-seeded random.Random, restricted to the shuffle operation used by
+ * option stability. Keep it private: changing the RNG changes which probes run. */
+export class StabilityRandom {
+ private readonly state = new Uint32Array(624);
+ private index = 624;
+
+ constructor(seedKey: string) {
+ let crc = 0xffffffff;
+ for (const byte of new TextEncoder().encode(seedKey)) {
+ crc ^= byte;
+ for (let bit = 0; bit < 8; bit++) crc = (crc >>> 1) ^ ((crc & 1) ? 0xedb88320 : 0);
+ }
+ const seed = (crc ^ 0xffffffff) >>> 0;
+ // CPython passes the integer's little-endian words to MT19937 init_by_array.
+ // CRC32 fits in one word; init_genrand(seed) alone produces different shuffles.
+ const mt = this.state;
+ mt[0] = 19650218;
+ for (let i = 1; i < mt.length; i++) {
+ mt[i] = (Math.imul(mt[i - 1] ^ (mt[i - 1] >>> 30), 1812433253) + i) >>> 0;
+ }
+ let i = 1;
+ for (let k = 624; k > 0; k--) {
+ mt[i] = ((mt[i] ^ Math.imul(mt[i - 1] ^ (mt[i - 1] >>> 30), 1664525)) + seed) >>> 0;
+ if (++i >= 624) { mt[0] = mt[623]; i = 1; }
+ }
+ for (let k = 623; k > 0; k--) {
+ mt[i] = ((mt[i] ^ Math.imul(mt[i - 1] ^ (mt[i - 1] >>> 30), 1566083941)) - i) >>> 0;
+ if (++i >= 624) { mt[0] = mt[623]; i = 1; }
+ }
+ mt[0] = 0x80000000;
+ }
+
+ private next(): number {
+ const mt = this.state;
+ if (this.index >= 624) {
+ for (let i = 0; i < 624; i++) {
+ const y = (mt[i] & 0x80000000) | (mt[(i + 1) % 624] & 0x7fffffff);
+ mt[i] = mt[(i + 397) % 624] ^ (y >>> 1) ^ ((y & 1) ? 0x9908b0df : 0);
+ }
+ this.index = 0;
+ }
+ let y = mt[this.index++];
+ y ^= y >>> 11;
+ y ^= (y << 7) & 0x9d2c5680;
+ y ^= (y << 15) & 0xefc60000;
+ y ^= y >>> 18;
+ return y >>> 0;
+ }
+
+ private below(n: number): number {
+ const bits = 32 - Math.clz32(n);
+ let r: number;
+ // random._randbelow uses n.bit_length(), including when n is a power of two,
+ // and consumes another word on rejection. A modulo would bias and desync it.
+ do { r = this.next() >>> (32 - bits); } while (r >= n);
+ return r;
+ }
+
+ shuffle<T>(values: T[]): void {
+ for (let i = values.length - 1; i > 0; i--) {
+ const j = this.below(i + 1);
+ [values[i], values[j]] = [values[j], values[i]];
+ }
+ }
+}
diff --git a/laya-ts/src/agent.ts b/laya-ts/src/agent.ts
index 1a7b4ec6..e6d96659 100644
--- a/laya-ts/src/agent.ts
+++ b/laya-ts/src/agent.ts
@@ -21,6 +21,7 @@ import {
} from "./common.js";
import type { BinningMap, InternalQ, MinConfidence, OptionStats, SequenceStats } from "./common.js";
import type { Batch, SessionProvider } from "./providers.js";
+import type { OptionStabilityReport } from "./option-stability.js";
import { encodeWithData, parseTokenizerJson, type TokenizerLike } from "./tokenizer.js";
import { decide, decideBatch, type DecideOptions, type DecideBatchOptions, type DecisionResult } from "./structured.js";
import {
@@ -56,6 +57,7 @@ export interface ChoiceAnswer {
answer_confidence: number;
action: ActionInfo;
low_confidence?: boolean;
+ reliability?: OptionStabilityReport;
}
export interface ScoreAnswer {
diff --git a/laya-ts/src/index.ts b/laya-ts/src/index.ts
index 21c9f921..cec8a0d0 100644
--- a/laya-ts/src/index.ts
+++ b/laya-ts/src/index.ts
@@ -37,6 +37,11 @@ export {
HOOK_EVENTS,
} from "./hooks.js";
export type { Hook, HookEvent, HookArg, PredictHook, PredictHookArg } from "./hooks.js";
+export { OptionStabilityHook, makeVariants, decideReliability, calibrateReliability } from "./option-stability.js";
+export type {
+ OptionStabilityOptions, OptionStabilityReport, StabilityProbe, ReliabilityDecision,
+ StabilityVariant, VariantOptions, CalibrationOptions,
+} from "./option-stability.js";
export { triageQuestions, emailQuestions, guardQuestions, moderationQuestions, routerQuestions } from "./presets.js";
export {
decide,
diff --git a/laya-ts/src/option-stability.ts b/laya-ts/src/option-stability.ts
new file mode 100644
index 00000000..2060fd3e
--- /dev/null
+++ b/laya-ts/src/option-stability.ts
@@ -0,0 +1,402 @@
+import { serializeState } from "./common.js";
+import { BaseHook, type PredictContext } from "./hooks.js";
+import { StabilityRandom } from "./_option-stability-random.js";
+
+export type StabilityProbe = "reorder" | "rename";
+export type ReliabilityDecision = "ACCEPT" | "VERIFY" | "ESCALATE";
+
+export interface StabilityVariant {
+ name: string;
+ order: string[];
+ relabel: boolean;
+}
+
+export interface VariantOptions {
+ nShuffles?: number;
+ relabel?: boolean;
+ include?: readonly string[] | null;
+ reorder?: boolean;
+}
+
+export interface CalibrationOptions {
+ targetAccuracy?: number;
+ maxEscalateAccuracy?: number;
+}
+
+export interface OptionStabilityOptions {
+ /** Domain-calibrated thresholds; pass null for uncalibrated metrics only. */
+ accept: number | null;
+ escalate: number | null;
+ probes?: "auto" | "all" | StabilityProbe | readonly StabilityProbe[] | null;
+ nShuffles?: number;
+ relabel?: boolean;
+ renameMaxOptions?: number;
+ variants?: "full" | "fast" | readonly string[] | null;
+ /** Use an explicit seed for cross-language parity with structured states. */
+ seed?: string | null;
+}
+
+export interface OptionStabilityReport {
+ probes: StabilityProbe[];
+ option_layout: "parallel" | "sequential";
+ decision: ReliabilityDecision | null;
+ /** The unrounded mean used by decideReliability and calibrateReliability. */
+ soft_stability: number | null;
+ stability: number | null;
+ rename_stability: number | null;
+ reorder_stability: number | null;
+ n_variants: number;
+ distinct_choices: string[];
+ variant_choices: Record<string, string | null>;
+ variant_support: Record<string, number>;
+}
+
+const SEP = "::rel";
+const ALL_VARIANTS = ["reversed", "shuffle1", "shuffle2", "shuffle3", "letters", "letters_reversed", "letters_shuffle"];
+
+/** Deterministic option permutations, including the original, with duplicates removed.
+ * The CRC32 / Python MT19937 sequence matches laya.hooks.make_variants. Filtering
+ * happens after shuffling so a named subset retains the full preset's permutations. */
+export function makeVariants(labels: readonly string[], seedKey: string, opts: VariantOptions = {}): StabilityVariant[] {
+ const { nShuffles = 3, relabel = true, include = null, reorder = true } = opts;
+ checkShuffles(nShuffles);
+ const rng = new StabilityRandom(seedKey);
+ const candidates: StabilityVariant[] = [{ name: "original", order: [...labels], relabel: false }];
+ if (reorder) {
+ candidates.push({ name: "reversed", order: [...labels].reverse(), relabel: false });
+ for (let i = 0; i < nShuffles; i++) {
+ const order = [...labels];
+ rng.shuffle(order);
+ candidates.push({ name: `shuffle${i + 1}`, order, relabel: false });
+ }
+ }
+ if (relabel && labels.length <= 26) {
+ candidates.push({ name: "letters", order: [...labels], relabel: true });
+ candidates.push({ name: "letters_reversed", order: [...labels].reverse(), relabel: true });
+ const order = [...labels];
+ rng.shuffle(order);
+ candidates.push({ name: "letters_shuffle", order, relabel: true });
+ }
+ const wanted = include === null ? null : new Set(include);
+ const seen = new Set<string>();
+ return candidates.filter((v) => {
+ if (wanted && v.name !== "original" && !wanted.has(v.name)) return false;
+ const key = JSON.stringify([v.order, v.relabel]);
+ if (seen.has(key)) return false;
+ seen.add(key);
+ return true;
+ });
+}
+
+/** Uncalibrated thresholds fail closed; equality with accept accepts, while equality
+ * with escalate verifies. A fitted accept of 1.01 accepts no probability score. */
+export function decideReliability(softStability: number, accept: number | null,
+ escalate: number | null = null): ReliabilityDecision | null {
+ checkProbability(softStability, "softStability");
+ checkThresholds(accept, escalate);
+ if (accept === null) return null;
+ if (softStability >= accept) return "ACCEPT";
+ if (escalate !== null && softStability < escalate) return "ESCALATE";
+ return "VERIFY";
+}
+
+/** Fit [accept, escalate] on labelled examples. Distinct raw scores are candidate
+ * cutoffs; equal scores stay together and report JSON can be replayed without rounding.
+ * Inputs must be equally sized, non-empty score/label arrays. */
+export function calibrateReliability(scores: readonly number[], correct: readonly boolean[],
+ opts: CalibrationOptions = {}): [number, number] {
+ const { targetAccuracy = 0.90, maxEscalateAccuracy = 1 / 3 } = opts;
+ if (scores.length !== correct.length) throw new Error("scores and correct must have the same length");
+ if (!scores.length) throw new Error("cannot calibrate reliability on empty scores/correct pairs");
+ checkProbability(targetAccuracy, "targetAccuracy");
+ checkProbability(maxEscalateAccuracy, "maxEscalateAccuracy");
+ const pairs = Array.from(scores, (score, i) => {
+ checkProbability(score, `scores[${i}]`);
+ if (typeof correct[i] !== "boolean") throw new TypeError(`correct[${i}] must be a bool`);
+ return [score, correct[i]] as const;
+ });
+ const cuts = [...new Set(pairs.map(([score]) => score))].sort((a, b) => a - b);
+ let accept = 1.01;
+ for (const t of cuts) {
+ const kept = pairs.filter(([s]) => s >= t);
+ if (kept.length && kept.filter(([, c]) => c).length / kept.length >= targetAccuracy) {
+ accept = t;
+ break;
+ }
+ }
+ let escalate = 0;
+ for (const t of cuts) {
+ if (t >= accept) break;
+ const below = pairs.filter(([s]) => s < t);
+ if (below.length && below.filter(([, c]) => c).length / below.length <= maxEscalateAccuracy) escalate = t;
+ }
+ return [accept, escalate];
+}
+
+function isRecord(value: unknown): value is Record<string, unknown> {
+ return value !== null && typeof value === "object" && !Array.isArray(value);
+}
+
+function checkShuffles(value: number): void {
+ if (typeof value !== "number" || !Number.isInteger(value) || value < 0) {
+ throw new Error("nShuffles must be a non-negative integer");
+ }
+}
+
+function checkThreshold(value: number | null, name: string, max: number): void {
+ if (value !== null && (typeof value !== "number" || !Number.isFinite(value) || value < 0 || value > max)) {
+ throw new Error(`${name} must be null or a number in [0, ${max}]`);
+ }
+}
+
+function checkProbability(value: number, name: string): void {
+ if (typeof value !== "number" || !Number.isFinite(value) || value < 0 || value > 1) {
+ throw new Error(`${name} must be a finite number in [0, 1]`);
+ }
+}
+
+function checkThresholds(accept: number | null, escalate: number | null): void {
+ checkThreshold(accept, "accept", 1.01);
+ checkThreshold(escalate, "escalate", 1);
+ if (accept !== null && escalate !== null && escalate > accept) {
+ throw new Error("escalate must not be greater than accept");
+ }
+}
+
+function resolveProbes(probes: OptionStabilityOptions["probes"]): StabilityProbe[] | null {
+ if (probes === undefined || probes === null || probes === "auto") return null;
+ if (probes === "all") return ["reorder", "rename"];
+ if (probes === "rename" || probes === "reorder") return [probes];
+ if (!Array.isArray(probes) || !probes.length || probes.some((p) => p !== "reorder" && p !== "rename")) {
+ throw new Error("probes must be 'auto', 'all', 'reorder', 'rename', or a non-empty sequence of probe names");
+ }
+ return [...new Set(probes)];
+}
+
+function resolveVariants(variants: OptionStabilityOptions["variants"]): string[] | null {
+ if (variants === undefined || variants === null || variants === "full") return null;
+ if (variants === "fast") return ["reversed", "letters_reversed"];
+ if (!Array.isArray(variants) || !variants.length || variants.some((v) => !ALL_VARIANTS.includes(v))) {
+ throw new Error(`variants must be 'full', 'fast', or a non-empty sequence from ${ALL_VARIANTS.join(", ")}`);
+ }
+ return [...variants];
+}
+
+function detectLayout(ctx: PredictContext): "parallel" | "sequential" {
+ const cfg = isRecord(ctx.agent) && isRecord(ctx.agent.cfg) ? ctx.agent.cfg : null;
+ const raw = cfg?.option_layout;
+ if (typeof raw === "string") {
+ const layout = raw.trim().toLowerCase();
+ if (layout === "parallel" || layout === "sequential") return layout;
+ console.warn(`laya: unknown option_layout ${JSON.stringify(raw)} in agent config; defaulting to 'sequential'`);
+ }
+ return "sequential";
+}
+
+function variantQuestion(q: Record<string, unknown>, order: string[], relabel: boolean):
+ { question: Record<string, unknown>; back: Map<string, string> } {
+ const shown = relabel ? order.map((_, i) => String.fromCharCode(65 + i)) : [...order];
+ const criteria = q.criteria;
+ const entries = shown.map((label, i) => {
+ const original = order[i];
+ let description = Array.isArray(criteria) ? null : (criteria as Record<string, unknown>)[original];
+ if (relabel && (description === null || description === undefined ||
+ (typeof description === "string" && description.trim() === ""))) description = original;
+ return [label, description] as const;
+ });
+ const nextCriteria = Object.fromEntries(entries);
+ const question = { ...q, criteria: Array.isArray(criteria) && !relabel ? [...order] : nextCriteria };
+ // Integer-like keys enumerate numerically in JS regardless of insertion order. The
+ // encoder already supports a slot permutation, so preserve the requested physical
+ // order there, including the caller's permutation of each generated variant.
+ const actual = Object.keys(nextCriteria);
+ const callerOrder = Array.isArray(q.option_order) ? q.option_order as number[] : order.map((_, i) => i);
+ const effectiveOrder = callerOrder.map((i) => actual.indexOf(shown[i]));
+ if (Object.hasOwn(q, "option_order") || effectiveOrder.some((i, slot) => i !== slot)) {
+ (question as Record<string, unknown>).option_order = effectiveOrder;
+ }
+ return { question, back: new Map(shown.map((label, i) => [label, order[i]])) };
+}
+
+/** Python round(value, 4): round the binary value itself, with ties to even.
+ * Multiplying a double by 1e4 first can manufacture a tie, and toFixed ties away. */
+function round4(value: number): number {
+ if (!Number.isFinite(value) || value === 0) return value;
+ const data = new DataView(new ArrayBuffer(8));
+ data.setFloat64(0, value);
+ const bits = data.getBigUint64(0);
+ const exponent = Number((bits >> 52n) & 0x7ffn);
+ const mantissa = (bits & ((1n << 52n) - 1n)) | (exponent ? 1n << 52n : 0n);
+ const power = exponent ? exponent - 1075 : -1074;
+ let numerator = mantissa * 10000n;
+ const denominator = power < 0 ? 1n << BigInt(-power) : 1n;
+ if (power > 0) numerator <<= BigInt(power);
+ let rounded = numerator / denominator;
+ const remainder = numerator % denominator;
+ if (remainder * 2n > denominator || (remainder * 2n === denominator && (rounded & 1n))) rounded++;
+ return (bits >> 63n ? -Number(rounded) : Number(rounded)) / 10000;
+}
+
+function compareCodePoints(a: string, b: string): number {
+ const x = Array.from(a, (c) => c.codePointAt(0)!);
+ const y = Array.from(b, (c) => c.codePointAt(0)!);
+ for (let i = 0; i < Math.min(x.length, y.length); i++) if (x[i] !== y[i]) return x[i] - y[i];
+ return x.length - y.length;
+}
+
+interface PlannedVariant {
+ name: string;
+ key: string;
+ back: Map<string, string>;
+ relabel: boolean;
+}
+interface QuestionPlan {
+ layout: "parallel" | "sequential";
+ variants: PlannedVariant[];
+ probes: StabilityProbe[];
+}
+
+/** Opt-in stability probes, batched with the original choice questions. Metrics are
+ * signals for calibration, not correctness probabilities. Layout detection selects
+ * probes; it does not add parallel-layout support to a provider. */
+export class OptionStabilityHook extends BaseHook {
+ private readonly accept: number | null;
+ private readonly escalate: number | null;
+ private readonly probes: StabilityProbe[] | null;
+ private readonly nShuffles: number;
+ private readonly relabel: boolean;
+ private readonly renameMaxOptions: number;
+ private readonly include: string[] | null;
+ private readonly seed: string | null;
+ private readonly plans = new WeakMap<PredictContext, Map<string, QuestionPlan>>();
+
+ constructor(opts: OptionStabilityOptions) {
+ super();
+ if (!opts || !Object.hasOwn(opts, "accept") || !Object.hasOwn(opts, "escalate")) {
+ throw new Error("accept and escalate are required; pass null for uncalibrated metrics");
+ }
+ checkThresholds(opts.accept, opts.escalate);
+ this.accept = opts.accept;
+ this.escalate = opts.escalate;
+ this.probes = resolveProbes(opts.probes);
+ this.nShuffles = opts.nShuffles === undefined ? 3 : opts.nShuffles;
+ checkShuffles(this.nShuffles);
+ this.relabel = opts.relabel === undefined ? true : opts.relabel;
+ if (typeof this.relabel !== "boolean") throw new TypeError("relabel must be a bool");
+ this.renameMaxOptions = opts.renameMaxOptions === undefined ? 20 : opts.renameMaxOptions;
+ if (!Number.isInteger(this.renameMaxOptions) || this.renameMaxOptions < 2) {
+ throw new Error("renameMaxOptions must be an integer >= 2");
+ }
+ this.include = resolveVariants(opts.variants);
+ this.seed = opts.seed === undefined || opts.seed === null ? null : String(opts.seed);
+ }
+
+ onPredictStart(ctx: PredictContext): void {
+ this.plans.delete(ctx);
+ for (const qid of Object.keys(ctx.questions)) {
+ if (qid.includes(SEP)) throw new Error(`question id ${JSON.stringify(qid)} must not contain ${JSON.stringify(SEP)}`);
+ }
+ const layout = detectLayout(ctx);
+ const configured = new Set(this.probes ?? (layout === "parallel"
+ ? (this.relabel ? ["rename"] : []) : (this.relabel ? ["reorder", "rename"] : ["reorder"])));
+ // Python uses repr for structured states; TS keeps its existing serializer.
+ // Explicit seed, string states, and multi-state "batch" seeds are shared.
+ const baseSeed = this.seed ?? (ctx.states.length === 1 ? serializeState(ctx.states[0]) : "batch");
+ const plan = new Map<string, QuestionPlan>();
+ const questions = { ...ctx.questions };
+ for (const [qid, q] of Object.entries(ctx.questions)) {
+ if (!isRecord(q) || q.type !== "choice") continue;
+ const criteria = q.criteria;
+ if (!Array.isArray(criteria) && !isRecord(criteria)) continue;
+ // Match the engine's list-to-dict answer keys, including numeric-like labels.
+ const labels = Object.keys(Array.isArray(criteria)
+ ? Object.fromEntries(criteria.map((label) => [String(label), null])) : criteria);
+ const variants: PlannedVariant[] = [{ name: "original", key: qid,
+ back: new Map(labels.map((label) => [label, label])), relabel: false }];
+ const probes: StabilityProbe[] = [];
+ if (labels.length >= 2) {
+ const generated = makeVariants(labels, `${baseSeed}|${qid}`, {
+ nShuffles: this.nShuffles,
+ relabel: configured.has("rename") && labels.length <= this.renameMaxOptions,
+ reorder: configured.has("reorder"), include: this.include,
+ }).slice(1);
+ if (generated.some((v) => !v.relabel)) probes.push("reorder");
+ if (generated.some((v) => v.relabel)) probes.push("rename");
+ generated.forEach((variant, i) => {
+ const key = `${qid}${SEP}${i + 1}`;
+ const { question, back } = variantQuestion(q, variant.order, variant.relabel);
+ questions[key] = question;
+ variants.push({ name: variant.name, key, back, relabel: variant.relabel });
+ });
+ }
+ plan.set(qid, { layout, variants, probes });
+ }
+ if (plan.size) {
+ ctx.questions = questions;
+ this.plans.set(ctx, plan);
+ }
+ }
+
+ onPredictEnd(ctx: PredictContext): void {
+ const plan = this.plans.get(ctx);
+ this.plans.delete(ctx);
+ if (!plan || !ctx.results) return;
+ const internalKeys = new Set([...plan.values()].flatMap((p) => p.variants.slice(1).map((v) => v.key)));
+ for (const result of ctx.results) {
+ if (!isRecord(result)) continue;
+ const answers = result.answers;
+ if (isRecord(answers)) {
+ for (const [qid, p] of plan) {
+ if (!Object.hasOwn(answers, qid)) continue;
+ const answer = answers[qid];
+ if (!isRecord(answer) || answer.type !== "choice" || typeof answer.choice !== "string") continue;
+ const seen: { name: string; choice: string | null; support: number; relabel: boolean }[] = [];
+ for (const variant of p.variants) {
+ if (!Object.hasOwn(answers, variant.key)) break;
+ const value = answers[variant.key];
+ if (!isRecord(value) || value.type !== "choice" || typeof value.choice !== "string" ||
+ !isRecord(value.probabilities)) break;
+ const probabilities = Object.entries(value.probabilities);
+ if (probabilities.some(([, v]) => typeof v !== "number" || !Number.isFinite(v) || v < 0 || v > 1)) break;
+ const probs = new Map(probabilities.map(([label, probability]) => [variant.back.get(label) ?? label, probability as number]));
+ const choice = variant.back.get(value.choice) ?? value.choice;
+ seen.push({ name: variant.name, choice, support: probs.get(answer.choice) ?? 0, relabel: variant.relabel });
+ }
+ // A partial or malformed answer set cannot establish stability.
+ if (seen.length !== p.variants.length) continue;
+ const others = seen.slice(1);
+ const probed = others.length > 0 && p.probes.length > 0;
+ const rate = (items: typeof seen): number | null => items.length
+ ? round4(items.filter((v) => v.choice === answer.choice).length / items.length) : null;
+ const soft = probed ? seen.reduce((sum, v) => sum + v.support, 0) / seen.length : null;
+ const report: OptionStabilityReport = {
+ probes: probed ? [...p.probes] : [], option_layout: p.layout,
+ decision: soft === null ? null : decideReliability(soft, this.accept, this.escalate),
+ soft_stability: soft, stability: probed ? rate(others) : null,
+ rename_stability: probed ? rate(others.filter((v) => v.relabel)) : null,
+ reorder_stability: probed ? rate(others.filter((v) => !v.relabel)) : null,
+ n_variants: seen.length,
+ distinct_choices: [...new Set(seen.flatMap((v) => v.choice === null ? [] : [v.choice]))].sort(compareCodePoints),
+ variant_choices: Object.fromEntries(seen.map((v) => [v.name, v.choice])),
+ variant_support: Object.fromEntries(seen.map((v) => [v.name, round4(v.support)])),
+ };
+ answer.reliability = report;
+ }
+ for (const key of internalKeys) delete answers[key];
+ }
+ const usage = result.usage;
+ if (isRecord(usage)) {
+ if (Array.isArray(usage.truncated_questions)) {
+ usage.truncated_questions = usage.truncated_questions.filter((key) => !internalKeys.has(key));
+ }
+ if (isRecord(usage.options)) {
+ usage.options = Object.fromEntries(Object.entries(usage.options).filter(([key]) => !internalKeys.has(key)));
+ }
+ }
+ }
+ }
+
+ onError(ctx: PredictContext): void {
+ this.plans.delete(ctx);
+ }
+}
diff --git a/laya-ts/src/router.ts b/laya-ts/src/router.ts
index acd38eb7..68d4e2b3 100644
--- a/laya-ts/src/router.ts
+++ b/laya-ts/src/router.ts
@@ -940,6 +940,7 @@ export class Router extends HookRegistry {
if (group.maxLen !== null) agentOpts.maxLen = group.maxLen;
if (group.headMaxLen !== null) agentOpts.headMaxLen = group.headMaxLen;
if (group.lang !== null) agentOpts.lang = group.lang;
+ markDefaultsRan(agentOpts);
let batchResults: SystemOneResult[];
if (typeof agent.predictBatch === "function") {
batchResults = await agent.predictBatch(
@@ -952,11 +953,13 @@ export class Router extends HookRegistry {
// still work; their states simply share no forward pass.
batchResults = [];
for (const [, ctx] of group.items) {
+ const singleOpts = group.lang === null ? {} : { lang: group.lang };
+ markDefaultsRan(singleOpts);
batchResults.push(
await agent.systemOne(
ctx.states[0],
group.questions as Record<string, QuestionDef>,
- group.lang === null ? undefined : { lang: group.lang },
+ singleOpts,
),
);
}
diff --git a/laya-ts/tests/option-stability-integration.test.ts b/laya-ts/tests/option-stability-integration.test.ts
new file mode 100644
index 00000000..bf0828b5
--- /dev/null
+++ b/laya-ts/tests/option-stability-integration.test.ts
@@ -0,0 +1,199 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import {
+ Agent, Router, OptionStabilityHook, PredictContext, clearDefaultHooks, defaultTokenizer,
+ setDefaultHooks, calibrateReliability, decideReliability, type Batch, type QuestionDef, type SessionProvider,
+} from "../src/index.js";
+
+afterEach(clearDefaultHooks);
+
+// Exercise real validation, encoding, collation, decoding and hook dispatch. Only the
+// model provider is synthetic: it selects the option span that still contains "approve".
+function semanticAgent(extra: Record<string, unknown> = {}) {
+ const batches: Batch[] = [];
+ const tok = {
+ ...defaultTokenizer(),
+ encode(text: string) {
+ return text.split(/[^\p{L}\p{N}_]+/u).filter(Boolean).map((word) => word === "approve" ? 900 : 901);
+ },
+ };
+ const provider: SessionProvider = {
+ async runEncoder(batch) {
+ batches.push(batch);
+ await Promise.resolve();
+ return { lastHidden: batch.inputIds.map(() => [[0]]) };
+ },
+ async runHead(_hidden, batch) {
+ const logits = batch.markerPos.map((positions, row) => positions.map((start, i) => {
+ if (!batch.markerMask[row][i]) return -100;
+ const next = positions[i + 1] || batch.inputIds[row].length;
+ const sep = batch.inputIds[row].indexOf(tok.sepId, start);
+ const end = sep < 0 ? next : Math.min(sep, next);
+ return batch.inputIds[row].slice(start, end).includes(900) ? 6 : 0;
+ }));
+ return { logits, act: batch.qtype.map(() => [3, 0]) };
+ },
+ };
+ return { agent: new Agent({ provider, tok, ...extra }), batches };
+}
+
+const hook = () => new OptionStabilityHook({ accept: 0.8, escalate: 0.6, variants: "fast", seed: "shared" });
+const question = (criteria: unknown, order?: number[]): Record<string, QuestionDef> => ({
+ action: { type: "choice", instructions: "Choose action", criteria, ...(order ? { option_order: order } : {}) },
+});
+
+describe("OptionStabilityHook through the local runtime", () => {
+ it("requires explicit finite thresholds for JavaScript callers too", () => {
+ for (const options of [{}, { accept: null }, { accept: undefined, escalate: null },
+ { accept: NaN, escalate: 0 }, { accept: 0.5, escalate: 0.6 }, { accept: true, escalate: null }]) {
+ expect(() => new OptionStabilityHook(options as never)).toThrow();
+ }
+ expect(() => decideReliability(0.9, undefined as never)).toThrow();
+ expect(decideReliability(0.9, null, null)).toBeNull();
+ expect(decideReliability(0.5, 0.8, null)).toBe("VERIFY");
+ });
+
+ it("rejects incomplete or invalid calibration evidence instead of silently dropping rows", () => {
+ expect(() => calibrateReliability([0.8, 0.9], [true])).toThrow();
+ expect(() => calibrateReliability([0.8], [true, false])).toThrow();
+ expect(() => calibrateReliability([], [])).toThrow();
+ expect(() => calibrateReliability([0.8], [1] as never)).toThrow();
+ for (const score of [NaN, Infinity, -0.1, 1.1]) {
+ expect(() => calibrateReliability([score], [true])).toThrow();
+ }
+ expect(() => calibrateReliability([0.8], [true], { targetAccuracy: NaN })).toThrow();
+ expect(() => calibrateReliability([0.8], [true], { maxEscalateAccuracy: 1.1 })).toThrow();
+ });
+
+ it("cleans incomplete or malformed probe results without manufacturing reliability", () => {
+ for (const bad of [undefined, null, { type: "choice", probabilities: { A: 1 } },
+ { type: "choice", choice: "A", probabilities: { A: NaN } },
+ { type: "choice", choice: "A", probabilities: { A: 1.1 } }]) {
+ const stability = hook();
+ const ctx = new PredictContext({ states: ["state"], questions: question(["approve", "reject"]) });
+ stability.onPredictStart(ctx);
+ const primary = { type: "choice", choice: "approve", probabilities: { approve: 1 } };
+ const answers = { action: primary, "action::rel1": bad, "action::rel2": bad };
+ ctx.results = [{ answers, usage: { options: { action: {}, "action::rel1": {} },
+ truncated_questions: ["action", "action::rel2"] } }];
+ expect(() => stability.onPredictEnd(ctx)).not.toThrow();
+ expect(answers).toEqual({ action: primary });
+ expect(primary).not.toHaveProperty("reliability");
+ expect(ctx.results[0].usage).toEqual({ options: { action: {} }, truncated_questions: ["action"] });
+ }
+ });
+
+ it.each([
+ ["list", ["approve", "reject", "wait"]],
+ ["empty descriptions", { approve: null, reject: "", wait: " " }],
+ ["mixed", { approve: "approve request", reject: null, wait: { action: "wait" } }],
+ ])("preserves %s semantics and caller inputs in a chunked batch", async (_name, criteria) => {
+ const q = question(criteria, [2, 0, 1]);
+ const original = structuredClone(q);
+ const events: string[] = [];
+ const { agent, batches } = semanticAgent({ hooks: [hook()] });
+ const results = await agent.predictBatch(["first", "second"], q, {
+ batchSize: 1,
+ async onPredictStart() { await Promise.resolve(); events.push("start"); },
+ async onPredictEnd(ctx) {
+ await Promise.resolve();
+ expect(Object.keys(ctx.results![0].answers as object)).toEqual(["action"]);
+ events.push("end");
+ },
+ });
+ expect(events).toEqual(["start", "end"]);
+ expect(q).toEqual(original);
+ expect(batches.map((b) => b.qtype.length)).toEqual([3, 3]);
+ for (const result of results) {
+ expect(Object.keys(result.answers)).toEqual(["action"]);
+ expect(result.answers.action).toMatchObject({ choice: "approve", reliability: {
+ decision: "ACCEPT", stability: 1, distinct_choices: ["approve"], n_variants: 3,
+ variant_choices: { original: "approve", reversed: "approve", letters_reversed: "approve" },
+ } });
+ expect(result.usage.truncated_questions?.some((id) => id.includes("::rel"))).toBe(false);
+ }
+ });
+
+ it("physically permutes integer-like labels and maps through caller option_order", async () => {
+ const stability = new OptionStabilityHook({ accept: null, escalate: null, variants: ["reversed"], seed: "n" });
+ const { agent, batches } = semanticAgent({ hooks: [stability] });
+ const result = await agent.predict("state", question({ "1": "reject", "2": "approve", "10": "wait" }, [1, 2, 0]));
+ expect(result.answers.action).toMatchObject({ choice: "2", reliability: { stability: 1, n_variants: 2 } });
+ // Original physical order is approve, wait, reject; reversed+caller order is approve,
+ // reject, wait. Non-semantic tokens coincide in the tokenizer, so inspect transformed
+ // questions separately below as well as checking the actual decoder's label.
+ expect(batches[0].markerPos).toHaveLength(2);
+ const ctx = new PredictContext({ states: ["state"], questions: question({ "1": "reject", "2": "approve", "10": "wait" }) });
+ stability.onPredictStart(ctx);
+ const variant = ctx.questions["action::rel1"] as QuestionDef;
+ expect(variant.option_order).toEqual([2, 1, 0]);
+ stability.onError(ctx);
+ });
+
+ it("keeps concurrent calls isolated when a hook instance is shared", async () => {
+ const stability = hook();
+ const { agent } = semanticAgent({ hooks: [stability] });
+ const [a, b] = await Promise.all([
+ agent.predict("one", question(["approve", "reject"])),
+ agent.predictBatch(["two", "three"], question(["wait", "approve", "reject"])),
+ ]);
+ for (const result of [a, ...b]) {
+ expect(result.answers.action).toMatchObject({ choice: "approve", reliability: { stability: 1 } });
+ expect(Object.keys(result.answers)).toEqual(["action"]);
+ }
+ });
+
+ it("does not manufacture probe evidence for a cached primary-only answer", async () => {
+ const stability = hook();
+ const { agent, batches } = semanticAgent({ hooks: [stability] });
+ const cached = { answers: { action: { type: "choice", choice: "approve", probabilities: { approve: 1 } } }, usage: {} };
+ const result = await agent.predict("cached", question(["approve", "reject"]), {
+ onPredictStart(ctx) { ctx.skip([structuredClone(cached)]); },
+ });
+ expect(result.answers.action).not.toHaveProperty("reliability");
+ expect(batches).toHaveLength(0);
+ expect((await agent.predict("fresh", question(["approve", "reject"]))).answers.action)
+ .toHaveProperty("reliability.stability", 1);
+ });
+
+ it.each(["provider", "validation", "timeout"])("cleans the plan after a %s failure and preserves the error", async (mode) => {
+ const stability = hook();
+ let captured: PredictContext | undefined;
+ const { agent } = semanticAgent({ hooks: [{ onPredictStart(ctx: PredictContext) { captured = ctx; } }, stability] });
+ if (mode === "provider") vi.spyOn(agent.provider, "runEncoder").mockRejectedValueOnce(new Error("provider failed"));
+ const q = question(["approve", "reject"]);
+ if (mode === "validation") q.action.option_order = [0, 0];
+ const opts = mode === "timeout" ? {
+ hooksTimeout: 0.001, onPredictStart: () => new Promise<void>(() => {}),
+ } : {};
+ await expect(agent.predict("state", q, opts)).rejects.toThrow(
+ mode === "provider" ? /provider failed/ : mode === "validation" ? /option_order/ : /exceeded/,
+ );
+ expect(captured!.error).not.toBeNull();
+ // Calling end again on the captured failed context must not find an old run plan.
+ const answers = Object.fromEntries(Object.entries(captured!.questions).map(([qid, definition]) => {
+ const criteria = (definition as QuestionDef).criteria;
+ const key = Array.isArray(criteria) ? criteria[0] : Object.keys(criteria as object)[0];
+ return [qid, { type: "choice", choice: key, probabilities: { [key]: 1 } }];
+ }));
+ captured!.results = [{ answers, usage: {} }];
+ stability.onPredictEnd(captured!);
+ expect(answers.action).not.toHaveProperty("reliability");
+ expect((await agent.predict("recovered", question(["approve", "reject"]))).answers.action)
+ .toHaveProperty("reliability.stability", 1);
+ });
+
+ it.each([false, true])("runs a default stability hook once through Router batch, fallback=%s", async (fallback) => {
+ const { agent } = semanticAgent();
+ const loaded = fallback ? { systemOne: agent.systemOne.bind(agent), cfg: agent.cfg } : agent;
+ const router = new Router({ default: "english", loader: (() => loaded) as never });
+ setDefaultHooks(hook());
+ const results = await router.predictBatch([
+ { state: "one", questions: question(["approve", "reject"]) },
+ { state: "two", questions: question(["approve", "reject"]) },
+ ]);
+ for (const result of results) {
+ expect(result.answers.action).toMatchObject({ choice: "approve", reliability: { stability: 1 } });
+ expect(Object.keys(result.answers)).toEqual(["action"]);
+ }
+ });
+});
diff --git a/laya-ts/tests/option-stability-parity.test.ts b/laya-ts/tests/option-stability-parity.test.ts
new file mode 100644
index 00000000..5389c859
--- /dev/null
+++ b/laya-ts/tests/option-stability-parity.test.ts
@@ -0,0 +1,158 @@
+import { readFileSync } from "node:fs";
+import { describe, expect, it } from "vitest";
+import {
+ OptionStabilityHook,
+ PredictContext,
+ calibrateReliability,
+ decideReliability,
+ makeVariants,
+ type CalibrationOptions,
+ type OptionStabilityOptions,
+ type OptionStabilityReport,
+ type StabilityVariant,
+ type VariantOptions,
+} from "../src/index.js";
+import { renderOptions } from "../src/common.js";
+
+type Question = {
+ type: "choice";
+ instructions: string;
+ criteria: string[] | Record<string, unknown>;
+ option_order?: number[];
+};
+type Result = {
+ answers: Record<string, {
+ type: "choice";
+ choice: string;
+ probabilities: Record<string, number>;
+ reliability?: OptionStabilityReport;
+ }>;
+ usage: Record<string, unknown>;
+};
+type HookCase = {
+ name: string;
+ states: unknown[];
+ questions: Record<string, Question>;
+ layout: "parallel" | "sequential";
+ options: OptionStabilityOptions;
+ expanded: { key: string; rendered: string[] }[];
+ model_results: Result[];
+ expected_results: Result[];
+};
+type Fixture = {
+ schema: string;
+ variant_cases: {
+ name: string; labels: string[]; seed_key: string; options: VariantOptions; expected: StabilityVariant[];
+ }[];
+ hook_cases: HookCase[];
+ calibration_cases: {
+ name: string; scores: number[]; correct: boolean[]; options: CalibrationOptions;
+ thresholds: [number, number]; decisions: (string | null)[];
+ }[];
+ decision_cases: { score: number; accept: number | null; escalate: number | null; expected: string | null }[];
+ pipeline: {
+ cases: string[]; correct: boolean[]; options: CalibrationOptions;
+ thresholds: [number, number]; decisions: (string | null)[];
+ };
+};
+
+// Generated by the actual Python hook and helpers, without model weights. The
+// fixture records the exact reference commits and file hashes, including pending
+// #1286/#1289 semantics. No Python installation is needed to run this TS suite.
+const fixture = JSON.parse(readFileSync(new URL("../fixtures/option-stability.json", import.meta.url), "utf8")) as Fixture;
+
+function physicalOptions(question: Question): string[] {
+ const criteria = Array.isArray(question.criteria)
+ ? Object.fromEntries(question.criteria.map((label) => [label, null]))
+ : question.criteria;
+ const rendered = renderOptions({ t: "choice", ins: question.instructions, crit: criteria });
+ return (question.option_order ?? rendered.map((_, index) => index)).map((index) => rendered[index]);
+}
+
+function runHook(c: HookCase): Result[] {
+ const questions = structuredClone(c.questions);
+ const hook = new OptionStabilityHook(c.options);
+ const context = new PredictContext({ states: structuredClone(c.states), questions,
+ agent: { cfg: { option_layout: c.layout } } });
+ hook.onPredictStart(context);
+ expect(questions).toEqual(c.questions);
+ // Compare what reaches each physical option slot. Integer-like JS object keys
+ // reorder themselves, so raw criteria insertion order is not a valid oracle;
+ // the TS hook uses option_order to realize the Python probe permutation.
+ expect(Object.entries(context.questions).map(([key, q]) => ({ key, rendered: physicalOptions(q as Question) })))
+ .toEqual(c.expanded);
+ context.results = structuredClone(c.model_results);
+ hook.onPredictEnd(context);
+ expect(context.results).toEqual(c.expected_results);
+ expect(questions).toEqual(c.questions);
+ // A completed plan must be consumed, so a repeated end event cannot apply the
+ // same mapping to already-cleaned output or produce an unprobed report.
+ hook.onPredictEnd(context);
+ expect(context.results).toEqual(c.expected_results);
+ return context.results as Result[];
+}
+
+describe("Python option stability fixtures", () => {
+ it("uses the supported fixture schema", () => {
+ expect(fixture.schema).toBe("laya-option-stability/1");
+ });
+
+ describe("CRC32 / Python integer-seeded shuffle / filtering / deduplication", () => {
+ for (const c of fixture.variant_cases) {
+ it(c.name, () => {
+ const labels = [...c.labels];
+ expect(makeVariants(labels, c.seed_key, c.options)).toEqual(c.expected);
+ expect(labels).toEqual(c.labels);
+ expect(makeVariants(labels, c.seed_key, c.options)).toEqual(c.expected);
+ });
+ }
+ });
+
+ describe("expanded semantics, label mapping, reporting and cleanup", () => {
+ for (const c of fixture.hook_cases) {
+ it(c.name, () => {
+ const results = runHook(c);
+ for (const result of results) {
+ const report = result.answers.q.reliability!;
+ if (report.soft_stability === null) {
+ expect(report.decision).toBeNull();
+ expect(report.probes).toEqual([]);
+ } else {
+ // Use the persisted public score, not rounded variant_support or an
+ // independently reconstructed mean, to replay the actual decision.
+ const saved = JSON.parse(JSON.stringify(report)) as OptionStabilityReport;
+ expect(decideReliability(saved.soft_stability!, c.options.accept, c.options.escalate))
+ .toBe(report.decision);
+ }
+ }
+ });
+ }
+ });
+
+ describe("raw calibration cohorts and boundary replay", () => {
+ for (const c of fixture.calibration_cases) {
+ it(c.name, () => {
+ const thresholds = calibrateReliability(c.scores, c.correct, c.options);
+ expect(thresholds).toEqual(c.thresholds);
+ expect(c.scores.map((score) => decideReliability(score, ...thresholds))).toEqual(c.decisions);
+ });
+ }
+ it("matches all categorical boundaries, including adjacent representable numbers", () => {
+ for (const c of fixture.decision_cases) {
+ expect(decideReliability(c.score, c.accept, c.escalate)).toBe(c.expected);
+ }
+ });
+ it("fits the persisted runtime reports and retains cuts that share four decimal digits", () => {
+ const pipeline = fixture.pipeline;
+ const scores = pipeline.cases.map((name) => {
+ const c = fixture.hook_cases.find((item) => item.name === name)!;
+ return runHook(c)[0].answers.q.reliability!.soft_stability!;
+ });
+ expect(new Set(scores).size).toBe(3);
+ const savedScores = JSON.parse(JSON.stringify(scores)) as number[];
+ const thresholds = calibrateReliability(savedScores, pipeline.correct, pipeline.options);
+ expect(thresholds).toEqual(pipeline.thresholds);
+ expect(savedScores.map((score) => decideReliability(score, ...thresholds))).toEqual(pipeline.decisions);
+ });
+ });
+});
diff --git a/laya-ts/tests/package-e2e.mjs b/laya-ts/tests/package-e2e.mjs
index ab60201a..3310ce38 100644
--- a/laya-ts/tests/package-e2e.mjs
+++ b/laya-ts/tests/package-e2e.mjs
@@ -107,7 +107,8 @@ test("the packed module installs and works in an external ESM TypeScript project
await writeFile(
join(consumerDir, "runtime.mjs"),
`import assert from "node:assert/strict";
-import { Agent, VERSION, predictTournament, DEFAULT_TOURNAMENT_GROUP } from "laya-ts";
+import { Agent, VERSION, predictTournament, DEFAULT_TOURNAMENT_GROUP,
+ OptionStabilityHook, calibrateReliability, decideReliability } from "laya-ts";
let encoderCalls = 0;
let headCalls = 0;
@@ -116,9 +117,9 @@ const provider = {
encoderCalls += 1;
return { lastHidden: [[[0]]] };
},
- async runHead() {
+ async runHead(_hidden, batch) {
headCalls += 1;
- return { logits: [[4, 0]], act: [[3, 0]] };
+ return { logits: batch.qtype.map(() => [4, 0]), act: batch.qtype.map(() => [3, 0]) };
},
};
const agent = new Agent({ provider });
@@ -142,6 +143,14 @@ const tournament = await predictTournament(agent, "charged twice", {
});
assert.equal(tournament.answers.intent.choice, "refund");
assert.deepEqual(tournament.tournament.intent, { labels: ["refund", "other"], n: 2, rounds: 0 });
+const stable = await agent.predict("charged twice", {
+ intent: { type: "choice", instructions: "Pick a route", criteria: ["refund", "other"] },
+}, { hooks: [new OptionStabilityHook({ accept: null, escalate: null, variants: "fast" })] });
+assert.deepEqual(Object.keys(stable.answers), ["intent"]);
+assert.equal(stable.answers.intent.reliability.n_variants, 3);
+assert.equal(stable.answers.intent.reliability.decision, null);
+const cuts = calibrateReliability([0.80001, 0.80002], [false, true], { targetAccuracy: 1 });
+assert.equal(decideReliability(0.80002, ...cuts), "ACCEPT");
`,
);
await run(process.execPath, ["runtime.mjs"], consumerDir);
@@ -153,6 +162,8 @@ assert.deepEqual(tournament.tournament.intent, { labels: ["refund", "other"], n:
VERSION,
predictTournament,
DEFAULT_TOURNAMENT_GROUP,
+ OptionStabilityHook,
+ type OptionStabilityReport,
type QuestionDef,
type SessionProvider,
type SystemOneResult,
@@ -187,6 +198,13 @@ void prediction;
const tournament: Promise<SystemOneResult & { tournament: Record<string, TournamentMeta> }> =
predictTournament(new Agent({ provider }), "charged twice", questions, DEFAULT_TOURNAMENT_GROUP);
void tournament;
+const stability = new OptionStabilityHook({ accept: null, escalate: null });
+new Agent({ provider }).addHook(stability);
+async function reliability(): Promise<OptionStabilityReport | undefined> {
+ const result = await prediction;
+ return result.answers.intent.type === "choice" ? result.answers.intent.reliability : undefined;
+}
+void reliability;
`,
);
await writeFile(
diff --git a/laya-ts/tests/router-batch.test.ts b/laya-ts/tests/router-batch.test.ts
index 74ecb7a0..a6e5ad78 100644
--- a/laya-ts/tests/router-batch.test.ts
+++ b/laya-ts/tests/router-batch.test.ts
@@ -1,10 +1,40 @@
import { describe, expect, it, vi } from "vitest";
-import { Agent, Router } from "../src/index.js";
+import { Agent, Router, clearDefaultHooks, setDefaultHooks } from "../src/index.js";
import type { BatchRequest } from "../src/index.js";
// TS mirror of tests/test_router_batch.py and the orchestration part of tests/test_batch.py.
const Q = { intent: { type: "noul", instructions: "Relevant?" } } as const;
+
+// A stateful default hook must not run again inside the Agent after the Router has
+// already expanded its questions (as OptionStabilityHook does).
+it.each([false, true])("runs process defaults once per request with fallback=%s", async (fallback) => {
+ const starts: unknown[] = [];
+ const ends: unknown[] = [];
+ const agent = new Agent({ provider: {
+ async runEncoder(batch) { return { lastHidden: batch.inputIds.map(() => [[0]]) }; },
+ async runHead(_hidden, batch) {
+ return { logits: batch.qtype.map(() => [2, 0]), act: batch.qtype.map(() => [3, 0]) };
+ },
+ } });
+ const loaded = fallback ? { systemOne: agent.systemOne.bind(agent) } : agent;
+ const router = new Router({ default: "english", loader: (() => loaded) as never });
+ setDefaultHooks({
+ onPredictStart(ctx) { starts.push([...ctx.states]); },
+ onPredictEnd(ctx) { ends.push([...ctx.states]); },
+ });
+ try {
+ const results = await router.predictBatch([
+ { state: "first request", questions: Q },
+ { state: "second request", questions: Q },
+ ]);
+ expect(results).toHaveLength(2);
+ expect(starts).toEqual([["first request"], ["second request"]]);
+ expect(ends).toEqual(starts);
+ } finally {
+ clearDefaultHooks();
+ }
+});
const req = (state: unknown, overrides: Record<string, unknown> = {}): BatchRequest =>
({ state, questions: Q, ...overrides }) as unknown as BatchRequest;
diff --git a/laya-ts/tests/types/option-stability.ts b/laya-ts/tests/types/option-stability.ts
new file mode 100644
index 00000000..24af48dd
--- /dev/null
+++ b/laya-ts/tests/types/option-stability.ts
@@ -0,0 +1,31 @@
+import {
+ Agent, OptionStabilityHook, calibrateReliability, decideReliability, makeVariants,
+ type OptionStabilityOptions, type OptionStabilityReport, type ReliabilityDecision,
+} from "../../src/index.js";
+
+declare const agent: Agent;
+const opts: OptionStabilityOptions = { accept: null, escalate: null, seed: "shared" };
+agent.addHook(new OptionStabilityHook(opts));
+const cuts: [number, number] = calibrateReliability([0.8, 0.9], [false, true], { targetAccuracy: 1 });
+const decision: ReliabilityDecision | null = decideReliability(0.9, ...cuts);
+const variants = makeVariants(["approve", "reject"], "seed", { nShuffles: 2 });
+const labels: string[] = variants[0].order;
+
+async function inspect() {
+ const result = await agent.predict("state", { q: { type: "choice", criteria: ["approve", "reject"] } });
+ if (result.answers.q.type === "choice") {
+ const report: OptionStabilityReport | undefined = result.answers.q.reliability;
+ const score: number | null | undefined = report?.soft_stability;
+ return score;
+ }
+}
+
+// @ts-expect-error explicit thresholds are required
+new OptionStabilityHook({ seed: "missing thresholds" });
+// @ts-expect-error thresholds must be numeric or null
+new OptionStabilityHook({ accept: "0.9", escalate: null });
+// @ts-expect-error invalid probe name
+new OptionStabilityHook({ accept: null, escalate: null, probes: "confidence" });
+void decision;
+void labels;
+void inspect;
diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.py
index 1a5840d9..772aff15 100644
--- a/tests/test_hooks_api.py
+++ b/tests/test_hooks_api.py
@@ -1594,6 +1594,14 @@ from laya import hooks as _hooks_mod # noqa: E402
check_true("hooks module exports compose_hooks", callable(_hooks_mod.compose_hooks))
+# The local TypeScript stability port consumes these reference helper contracts.
+check("stability/make_variants reference parameters", list(sig(_hooks_mod.make_variants)),
+ ["labels", "seed_key", "n_shuffles", "relabel", "include", "reorder"])
+check("stability/calibrate_reliability reference parameters", list(sig(_hooks_mod.calibrate_reliability)),
+ ["soft_scores", "correct", "target_accuracy", "max_escalate_accuracy"])
+check("stability/decide_reliability reference parameters", list(sig(_hooks_mod.decide_reliability)),
+ ["soft_stability", "accept", "escalate"])
+
_patterns_md = os.path.join(REPO, "docs", "hooks", "patterns.md")
with open(_patterns_md, encoding="utf-8", newline="") as _pmf:
_patterns_text = _pmf.read().replace("\r\n", "\n")