Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(ts): add local option stability diagnostics
xpff8d906e8480
diff --git a/laya-ts/README.md b/laya-ts/README.mdindex 627e3c93..4451768c 100644--- a/laya-ts/README.md+++ b/laya-ts/README.md@@ -75,6 +75,76 @@ class MetricsHook extends BaseHook {setDefaultHooks([new MetricsHook()]); // addDefaultHook(...) appends; clearDefaultHooks() resets```+### Option stability++`OptionStabilityHook` adds opt-in `reliability` diagnostics to choice answers. It expands+the questions with reorder and letter-rename probes and evaluates them through the existing+batch path. The original answer stays in place; internal probe answers and their usage keys+are removed before the result returns. Extra rows cost inference work even within one call.++```ts+import { Agent, OptionStabilityHook, calibrateReliability, decideReliability } from "laya-ts";++const agent = await Agent.load("./dist/laya");+agent.addHook(new OptionStabilityHook({+ accept: null, escalate: null, variants: "fast", seed: "evaluation-v1",+}));+const result = await agent.predict("Please approve the request", {+ action: { type: "choice", instructions: "Choose action", criteria: ["approve", "reject"] },+});+if (result.answers.action.type === "choice") {+ console.log(result.answers.action.reliability);+}++// Fit on your labelled examples, using only probed numeric soft_stability values.+// Keep these scores and cutoffs as numbers; round only a separate display string.+function fitAndReplay(scores: number[], correct: boolean[], savedScore: number) {+ const [accept, escalate] = calibrateReliability(scores, correct, { targetAccuracy: 0.9 });+ return decideReliability(savedScore, accept, escalate);+}+```++Both thresholds are required. `accept: null` disables categorical decisions;+`escalate: null` disables only `ESCALATE`. No calibrated defaults are recommended.+`ACCEPT` uses `soft_stability >= accept`; `ESCALATE` uses+`soft_stability < escalate`; otherwise the decision is `VERIFY`. An unprobed question has+null signals. The report keeps the unrounded mean used by the comparator, matching the+Python precision correction in #1289. Other diagnostic fractions retain Python's+four-decimal rounding. Historical rounded Python reports cannot recover the lost precision.++Calibration requires equally sized, nonempty score/boolean-label arrays, finite scores in+`[0, 1]`, and accuracy targets in `[0, 1]`. Invalid evidence is rejected rather than silently+dropping unmatched rows. The helpers match Python on valid inputs; they do not copy its+permissive handling of invalid calibration inputs.++Options use camelCase (`nShuffles`, `renameMaxOptions`); result fields match Python's+snake_case fields. `variants` accepts `"fast"`, `"full"`, or an array of variant names;+`probes` accepts `"auto"`, `"reorder"`, `"rename"`, `"all"`, or a probe-name array.+Renaming retains original label text for missing descriptions, including blank strings,+as in #1286, and preserves nonblank descriptions and caller `option_order`.++Explicit string seeds and the same ordered labels produce the same permutations as Python's+CRC32-seeded shuffle. JavaScript enumerates integer-like object keys numerically, so preserve+the intended label order when comparing inputs across languages.+They reproduce probe selection, not model outputs. The default seed uses the single string+state, or `"batch"` for multiple states. A single structured state uses the existing TS state+serializer; Python uses `repr`, so supply the same explicit seed for cross-language replay.+The shared fixtures record their Python version and source hashes. Numerical comparison is+for those decoded inputs; different inference backends or Python floating-point accumulation+versions can still produce different scores near a threshold.++The hook reads `agent.cfg.option_layout` to choose probes automatically: a declared+parallel layout selects rename-only probes; sequential selects both. This is probe selection,+not an implementation of parallel attention. The current local ONNX runtime supports the+sequential path; this hook adds no new checkpoint/backend support. It also does not add the+feature to the separate HTTP SDK.++Install the hook once at the Agent, Router, or process-default scope. `predictBatch` and+concurrent calls can share one instance; plans are scoped to each prediction context and+released on completion or error. A cache hit without probe answers does not gain a fabricated+reliability score. The hook's own methods are synchronous and work within the existing async+Agent/Router prediction lifecycle.+## Structured decisions (`decide`)Turn a JSON schema into typed values in one call — the port of Python's `laya.structured`diff --git a/laya-ts/fixtures/option-stability.json b/laya-ts/fixtures/option-stability.jsonnew file mode 100644index 00000000..aa988871--- /dev/null+++ b/laya-ts/fixtures/option-stability.json@@ -0,0 +1,128 @@+{+ "schema": "laya-option-stability/1",+ "reference": {+ "base_commit": "ffc142012af63d015fa19f05f29f9c5ff329a2cc",+ "overlays": [+ {+ "commit": "8257e91e4c8c56902720496de75c0a8e2f3f6be7",+ "pull_request": "https://github.com/NandhaKishorM/laya/pull/1289"+ },+ {+ "commit": "7feac1d3ba4e30ba09bf658f0ec15001b64a9223",+ "pull_request": "https://github.com/NandhaKishorM/laya/pull/1286"+ }+ ],+ "method": "git show precision commit source; git apply exact rename commit patch for laya/hooks.py; local combined reference, not upstream main",+ "files": {+ "laya/hooks.py": "b8f34a77573146b8d6fcdf3d3bc384fac73c52ac612e8fec7218dc3ae9de9321",+ "laya/common.py": "8396e8c8dbceabfc7fb2abb1e2fe4692751389ba9bd80d592f1ea922fef0765c"+ },+ "python": "3.11.13"+ },+ "variant_cases": [+ {"name":"two-dedup","labels":["yes","no"],"seed_key":"","options":{},"expected":[{"name":"original","order":["yes","no"],"relabel":false},{"name":"reversed","order":["no","yes"],"relabel":false},{"name":"letters","order":["yes","no"],"relabel":true},{"name":"letters_reversed","order":["no","yes"],"relabel":true}]},+ {"name":"three-utf8","labels":["alpha","beta","gamma"],"seed_key":"种子|café|🙂","options":{},"expected":[{"name":"original","order":["alpha","beta","gamma"],"relabel":false},{"name":"reversed","order":["gamma","beta","alpha"],"relabel":false},{"name":"shuffle2","order":["beta","alpha","gamma"],"relabel":false},{"name":"shuffle3","order":["beta","gamma","alpha"],"relabel":false},{"name":"letters","order":["alpha","beta","gamma"],"relabel":true},{"name":"letters_reversed","order":["gamma","beta","alpha"],"relabel":true},{"name":"letters_shuffle","order":["alpha","gamma","beta"],"relabel":true}]},+ {"name":"twenty-zero-shuffles","labels":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19"],"seed_key":"twenty","options":{"nShuffles":0},"expected":[{"name":"original","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19"],"relabel":false},{"name":"reversed","order":["v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":false},{"name":"letters","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19"],"relabel":true},{"name":"letters_reversed","order":["v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":true},{"name":"letters_shuffle","order":["v14","v5","v1","v11","v8","v6","v17","v12","v9","v7","v0","v3","v4","v15","v10","v16","v2","v13","v18","v19"],"relabel":true}]},+ {"name":"twenty-six-twenty-shuffles","labels":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25"],"seed_key":"twenty-six","options":{"nShuffles":20},"expected":[{"name":"original","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25"],"relabel":false},{"name":"reversed","order":["v25","v24","v23","v22","v21","v20","v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":false},{"name":"shuffle1","order":["v10","v19","v9","v25","v11","v20","v0","v6","v24","v12","v5","v18","v14","v3","v7","v17","v1","v4","v2","v13","v8","v23","v15","v21","v16","v22"],"relabel":false},{"name":"shuffle2","order":["v14","v8","v19","v0","v21","v18","v16","v22","v4","v6","v5","v2","v3","v7","v17","v24","v20","v11","v13","v9","v1","v10","v12","v25","v15","v23"],"relabel":false},{"name":"shuffle3","order":["v8","v18","v16","v23","v7","v5","v17","v22","v3","v19","v12","v9","v6","v4","v11","v0","v1","v15","v25","v20","v10","v14","v13","v24","v21","v2"],"relabel":false},{"name":"shuffle4","order":["v5","v17","v11","v10","v9","v3","v14","v23","v2","v21","v4","v16","v13","v19","v18","v7","v6","v24","v12","v25","v22","v0","v8","v20","v1","v15"],"relabel":false},{"name":"shuffle5","order":["v16","v6","v22","v12","v25","v2","v5","v1","v24","v7","v19","v17","v9","v10","v21","v20","v8","v15","v23","v14","v4","v11","v18","v13","v0","v3"],"relabel":false},{"name":"shuffle6","order":["v7","v16","v24","v11","v1","v15","v22","v14","v20","v21","v5","v2","v0","v6","v3","v8","v10","v13","v23","v18","v4","v19","v25","v9","v12","v17"],"relabel":false},{"name":"shuffle7","order":["v21","v20","v16","v24","v2","v25","v12","v17","v6","v15","v23","v14","v1","v8","v10","v9","v11","v0","v4","v5","v13","v3","v19","v18","v22","v7"],"relabel":false},{"name":"shuffle8","order":["v25","v4","v12","v20","v19","v16","v10","v6","v9","v14","v1","v18","v24","v3","v22","v0","v5","v15","v21","v8","v2","v23","v11","v13","v7","v17"],"relabel":false},{"name":"shuffle9","order":["v3","v2","v1","v0","v11","v20","v24","v12","v17","v8","v18","v7","v16","v5","v21","v14","v22","v6","v10","v19","v15","v13","v25","v4","v9","v23"],"relabel":false},{"name":"shuffle10","order":["v18","v17","v21","v19","v3","v24","v1","v20","v0","v9","v12","v22","v11","v15","v8","v16","v5","v14","v2","v23","v13","v25","v6","v4","v7","v10"],"relabel":false},{"name":"shuffle11","order":["v2","v24","v8","v19","v17","v15","v13","v16","v5","v11","v23","v9","v10","v14","v25","v21","v1","v12","v22","v7","v4","v20","v3","v0","v18","v6"],"relabel":false},{"name":"shuffle12","order":["v8","v12","v13","v11","v24","v6","v25","v22","v0","v23","v9","v1","v2","v5","v4","v19","v17","v20","v21","v3","v7","v14","v18","v16","v15","v10"],"relabel":false},{"name":"shuffle13","order":["v24","v22","v25","v5","v16","v20","v17","v4","v7","v3","v12","v19","v2","v11","v14","v9","v10","v0","v21","v15","v6","v1","v13","v23","v18","v8"],"relabel":false},{"name":"shuffle14","order":["v17","v20","v21","v22","v16","v11","v0","v10","v14","v25","v4","v13","v8","v9","v15","v19","v1","v7","v5","v24","v23","v6","v3","v12","v2","v18"],"relabel":false},{"name":"shuffle15","order":["v13","v4","v19","v10","v2","v11","v7","v1","v6","v3","v24","v23","v18","v9","v8","v16","v17","v20","v21","v15","v5","v14","v12","v22","v0","v25"],"relabel":false},{"name":"shuffle16","order":["v11","v12","v17","v7","v15","v13","v14","v10","v20","v0","v22","v23","v16","v21","v4","v24","v1","v6","v2","v25","v3","v8","v19","v9","v5","v18"],"relabel":false},{"name":"shuffle17","order":["v2","v8","v4","v3","v22","v14","v20","v12","v5","v7","v17","v23","v13","v15","v21","v16","v10","v0","v24","v18","v19","v1","v11","v6","v25","v9"],"relabel":false},{"name":"shuffle18","order":["v6","v1","v14","v25","v2","v22","v3","v20","v8","v23","v18","v0","v4","v7","v13","v17","v21","v5","v11","v19","v24","v16","v12","v15","v10","v9"],"relabel":false},{"name":"shuffle19","order":["v15","v3","v12","v16","v7","v2","v6","v8","v10","v18","v5","v22","v24","v20","v19","v23","v14","v11","v0","v9","v1","v25","v17","v4","v13","v21"],"relabel":false},{"name":"shuffle20","order":["v7","v17","v1","v5","v16","v25","v11","v12","v15","v18","v2","v4","v6","v23","v14","v19","v24","v13","v22","v10","v21","v3","v8","v20","v9","v0"],"relabel":false},{"name":"letters","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25"],"relabel":true},{"name":"letters_reversed","order":["v25","v24","v23","v22","v21","v20","v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":true},{"name":"letters_shuffle","order":["v1","v3","v22","v8","v2","v11","v5","v4","v15","v14","v20","v21","v12","v16","v24","v18","v13","v10","v9","v7","v25","v23","v19","v6","v17","v0"],"relabel":true}]},+ {"name":"twenty-seven-no-letters","labels":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25","v26"],"seed_key":"twenty-seven","options":{},"expected":[{"name":"original","order":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25","v26"],"relabel":false},{"name":"reversed","order":["v26","v25","v24","v23","v22","v21","v20","v19","v18","v17","v16","v15","v14","v13","v12","v11","v10","v9","v8","v7","v6","v5","v4","v3","v2","v1","v0"],"relabel":false},{"name":"shuffle1","order":["v4","v11","v0","v16","v25","v8","v15","v2","v13","v5","v23","v20","v26","v24","v6","v14","v22","v21","v3","v19","v7","v12","v9","v18","v1","v17","v10"],"relabel":false},{"name":"shuffle2","order":["v1","v4","v23","v20","v8","v6","v12","v14","v9","v7","v13","v16","v0","v21","v17","v25","v11","v10","v2","v19","v15","v24","v22","v26","v18","v5","v3"],"relabel":false},{"name":"shuffle3","order":["v5","v6","v0","v8","v24","v17","v19","v22","v4","v23","v18","v11","v15","v20","v12","v2","v16","v9","v10","v1","v13","v21","v26","v7","v14","v3","v25"],"relabel":false}]},+ {"name":"fast-include","labels":["a","b","c"],"seed_key":"seed","options":{"include":["reversed","letters_reversed"]},"expected":[{"name":"original","order":["a","b","c"],"relabel":false},{"name":"reversed","order":["c","b","a"],"relabel":false},{"name":"letters_reversed","order":["c","b","a"],"relabel":true}]},+ {"name":"custom-include","labels":["a","b","c","d"],"seed_key":"seed","options":{"include":["shuffle3","letters_shuffle"]},"expected":[{"name":"original","order":["a","b","c","d"],"relabel":false},{"name":"shuffle3","order":["b","d","a","c"],"relabel":false},{"name":"letters_shuffle","order":["a","d","c","b"],"relabel":true}]},+ {"name":"rename-only","labels":["a","b","c"],"seed_key":"seed","options":{"reorder":false},"expected":[{"name":"original","order":["a","b","c"],"relabel":false},{"name":"letters","order":["a","b","c"],"relabel":true},{"name":"letters_reversed","order":["c","b","a"],"relabel":true}]},+ {"name":"reorder-only","labels":["a","b","c"],"seed_key":"seed","options":{"relabel":false},"expected":[{"name":"original","order":["a","b","c"],"relabel":false},{"name":"reversed","order":["c","b","a"],"relabel":false},{"name":"shuffle2","order":["a","c","b"],"relabel":false}]},+ {"name":"no-probes","labels":["a","b","c"],"seed_key":"seed","options":{"relabel":false,"reorder":false},"expected":[{"name":"original","order":["a","b","c"],"relabel":false}]},+ {"name":"filtered-away","labels":["a","b","c"],"seed_key":"seed","options":{"nShuffles":0,"include":["shuffle3"]},"expected":[{"name":"original","order":["a","b","c"],"relabel":false}]},+ {"name":"unicode-labels","labels":["","🙂","é"],"seed_key":"nul\u0000seed","options":{"nShuffles":5},"expected":[{"name":"original","order":["","🙂","é"],"relabel":false},{"name":"reversed","order":["é","🙂",""],"relabel":false},{"name":"shuffle1","order":["","é","🙂"],"relabel":false},{"name":"shuffle2","order":["é","","🙂"],"relabel":false},{"name":"shuffle5","order":["🙂","","é"],"relabel":false},{"name":"letters","order":["","🙂","é"],"relabel":true},{"name":"letters_reversed","order":["é","🙂",""],"relabel":true},{"name":"letters_shuffle","order":["🙂","é",""],"relabel":true}]}+ ],+ "hook_cases": [+ {"name":"described-full","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["review: Review the request","accept: Accept the request","reject: Reject the request"]},{"key":"q::rel3","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel4","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]},{"key":"q::rel5","rendered":["A: Review the request","B: Accept the request","C: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}},"q::rel3":{"type":"choice","choice":"A","probabilities":{"A":0.8,"B":0.19999999999999996,"C":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"C":0.25,"B":0.75,"A":0.0}},"q::rel5":{"type":"choice","choice":"B","probabilities":{"B":0.7,"A":0.30000000000000004,"C":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.6667,"reorder_stability":0.5,"n_variants":6,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","shuffle2":"accept","letters":"accept","letters_reversed":"review","letters_shuffle":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"letters":0.8,"letters_reversed":0.25,"letters_shuffle":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"list-fast","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":["accept","review","reject"]}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["accept","review","reject"]},{"key":"q::rel1","rendered":["reject","review","accept"]},{"key":"q::rel2","rendered":["A: reject","B: review","C: accept"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"undescribed","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":null,"review":"","reject":" \t\n"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["accept","review","reject: \t\n"]},{"key":"q::rel1","rendered":["reject: \t\n","review","accept"]},{"key":"q::rel2","rendered":["A: reject","B: review","C: accept"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"mixed-values","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"missing":null,"blank":" ","zero":0,"false":false,"object":{"detail":"café","priority":2},"array":["x",3]}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["missing","blank: ","zero: 0","false: false","object: {\"detail\": \"café\", \"priority\": 2}","array: [\"x\", 3]"]},{"key":"q::rel1","rendered":["array: [\"x\", 3]","object: {\"detail\": \"café\", \"priority\": 2}","false: false","zero: 0","blank: ","missing"]},{"key":"q::rel2","rendered":["A: [\"x\", 3]","B: {\"detail\": \"café\", \"priority\": 2}","C: false","D: 0","E: blank","F: missing"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"missing","probabilities":{"missing":0.8,"blank":0.19999999999999996,"zero":0.0,"false":0.0,"object":0.0,"array":0.0}},"q::rel1":{"type":"choice","choice":"blank","probabilities":{"missing":0.25,"blank":0.75,"zero":0.0,"false":0.0,"object":0.0,"array":0.0}},"q::rel2":{"type":"choice","choice":"F","probabilities":{"F":0.7,"E":0.30000000000000004,"D":0.0,"C":0.0,"B":0.0,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":6,"distinct":6},"q::rel1":{"total":6,"distinct":6},"q::rel2":{"total":6,"distinct":6}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"missing","probabilities":{"missing":0.8,"blank":0.19999999999999996,"zero":0.0,"false":0.0,"object":0.0,"array":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["blank","missing"],"variant_choices":{"original":"missing","reversed":"blank","letters_reversed":"missing"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":6,"distinct":6}}}}]},+ {"name":"numeric-keys-order","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"1":"first","2":"second","10":"tenth"},"option_order":[2,0,1]}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["10: tenth","1: first","2: second"]},{"key":"q::rel1","rendered":["1: first","10: tenth","2: second"]},{"key":"q::rel2","rendered":["10: tenth","2: second","1: first"]},{"key":"q::rel3","rendered":["C: tenth","A: first","B: second"]},{"key":"q::rel4","rendered":["C: first","A: tenth","B: second"]},{"key":"q::rel5","rendered":["C: tenth","A: second","B: first"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"1","probabilities":{"1":0.8,"2":0.19999999999999996,"10":0.0}},"q::rel1":{"type":"choice","choice":"2","probabilities":{"1":0.25,"2":0.75,"10":0.0}},"q::rel2":{"type":"choice","choice":"1","probabilities":{"1":0.7,"2":0.30000000000000004,"10":0.0}},"q::rel3":{"type":"choice","choice":"A","probabilities":{"A":0.8,"B":0.19999999999999996,"C":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"C":0.25,"B":0.75,"A":0.0}},"q::rel5":{"type":"choice","choice":"B","probabilities":{"B":0.7,"A":0.30000000000000004,"C":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"1","probabilities":{"1":0.8,"2":0.19999999999999996,"10":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.6667,"reorder_stability":0.5,"n_variants":6,"distinct_choices":["1","2"],"variant_choices":{"original":"1","reversed":"2","shuffle2":"1","letters":"1","letters_reversed":"2","letters_shuffle":"1"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"letters":0.8,"letters_reversed":0.25,"letters_shuffle":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"numeric-list","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":["1","2","10"]}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["1","2","10"]},{"key":"q::rel1","rendered":["10","2","1"]},{"key":"q::rel2","rendered":["A: 10","B: 2","C: 1"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"1","probabilities":{"1":0.8,"2":0.19999999999999996,"10":0.0}},"q::rel1":{"type":"choice","choice":"2","probabilities":{"1":0.25,"2":0.75,"10":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"1","probabilities":{"1":0.8,"2":0.19999999999999996,"10":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["1","2"],"variant_choices":{"original":"1","reversed":"2","letters_reversed":"1"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"prototype-labels","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"__proto__":"base","constructor":"create","toString":"print"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["__proto__: base","constructor: create","toString: print"]},{"key":"q::rel1","rendered":["toString: print","constructor: create","__proto__: base"]},{"key":"q::rel2","rendered":["constructor: create","__proto__: base","toString: print"]},{"key":"q::rel3","rendered":["A: base","B: create","C: print"]},{"key":"q::rel4","rendered":["A: print","B: create","C: base"]},{"key":"q::rel5","rendered":["A: create","B: base","C: print"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"__proto__","probabilities":{"__proto__":0.8,"constructor":0.19999999999999996,"toString":0.0}},"q::rel1":{"type":"choice","choice":"constructor","probabilities":{"__proto__":0.25,"constructor":0.75,"toString":0.0}},"q::rel2":{"type":"choice","choice":"__proto__","probabilities":{"__proto__":0.7,"constructor":0.30000000000000004,"toString":0.0}},"q::rel3":{"type":"choice","choice":"A","probabilities":{"A":0.8,"B":0.19999999999999996,"C":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"C":0.25,"B":0.75,"A":0.0}},"q::rel5":{"type":"choice","choice":"B","probabilities":{"B":0.7,"A":0.30000000000000004,"C":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"__proto__","probabilities":{"__proto__":0.8,"constructor":0.19999999999999996,"toString":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.6667,"reorder_stability":0.5,"n_variants":6,"distinct_choices":["__proto__","constructor"],"variant_choices":{"original":"__proto__","reversed":"constructor","shuffle2":"__proto__","letters":"__proto__","letters_reversed":"constructor","letters_shuffle":"__proto__"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"letters":0.8,"letters_reversed":0.25,"letters_shuffle":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"unicode-labels","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"":"private","🙂":"smile","é":"accent"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":[": private","🙂: smile","é: accent"]},{"key":"q::rel1","rendered":["é: accent","🙂: smile",": private"]},{"key":"q::rel2","rendered":["🙂: smile",": private","é: accent"]},{"key":"q::rel3","rendered":["A: private","B: smile","C: accent"]},{"key":"q::rel4","rendered":["A: accent","B: smile","C: private"]},{"key":"q::rel5","rendered":["A: smile","B: private","C: accent"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"","probabilities":{"":0.8,"🙂":0.19999999999999996,"é":0.0}},"q::rel1":{"type":"choice","choice":"🙂","probabilities":{"":0.25,"🙂":0.75,"é":0.0}},"q::rel2":{"type":"choice","choice":"","probabilities":{"":0.7,"🙂":0.30000000000000004,"é":0.0}},"q::rel3":{"type":"choice","choice":"A","probabilities":{"A":0.8,"B":0.19999999999999996,"C":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"C":0.25,"B":0.75,"A":0.0}},"q::rel5":{"type":"choice","choice":"B","probabilities":{"B":0.7,"A":0.30000000000000004,"C":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"","probabilities":{"":0.8,"🙂":0.19999999999999996,"é":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.6667,"reorder_stability":0.5,"n_variants":6,"distinct_choices":["","🙂"],"variant_choices":{"original":"","reversed":"🙂","shuffle2":"","letters":"","letters_reversed":"🙂","letters_shuffle":""},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"letters":0.8,"letters_reversed":0.25,"letters_shuffle":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"custom-variants","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":["shuffle3","letters_shuffle"],"seed":"种子🙂"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["A: Review the request","B: Accept the request","C: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"A","probabilities":{"B":0.25,"A":0.75,"C":0.0}}},"usage":{"input_tokens":200,"output_tokens":2,"truncated_questions":["q","q::rel1"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.525,"stability":0.0,"rename_stability":0.0,"reorder_stability":null,"n_variants":2,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","letters_shuffle":"review"},"variant_support":{"original":0.8,"letters_shuffle":0.25}}}},"usage":{"input_tokens":200,"output_tokens":2,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"zero-shuffles","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"nShuffles":0},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel3","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"A","probabilities":{"A":0.7,"B":0.30000000000000004,"C":0.0}},"q::rel3":{"type":"choice","choice":"C","probabilities":{"C":0.8,"B":0.19999999999999996,"A":0.0}}},"usage":{"input_tokens":400,"output_tokens":4,"truncated_questions":["q","q::rel1","q::rel2","q::rel3"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.6375,"stability":0.6667,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":4,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters":"accept","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters":0.7,"letters_reversed":0.8}}}},"usage":{"input_tokens":400,"output_tokens":4,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"extra-shuffles","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"nShuffles":5},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["review: Review the request","accept: Accept the request","reject: Reject the request"]},{"key":"q::rel3","rendered":["review: Review the request","reject: Reject the request","accept: Accept the request"]},{"key":"q::rel4","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel5","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]},{"key":"q::rel6","rendered":["A: Review the request","B: Accept the request","C: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}},"q::rel3":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"A":0.25,"B":0.75,"C":0.0}},"q::rel5":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}},"q::rel6":{"type":"choice","choice":"B","probabilities":{"B":0.8,"A":0.19999999999999996,"C":0.0}}},"usage":{"input_tokens":700,"output_tokens":7,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5","q::rel6"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3},"q::rel6":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.6142857142857142,"stability":0.6667,"rename_stability":0.6667,"reorder_stability":0.6667,"n_variants":7,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","shuffle2":"accept","shuffle5":"accept","letters":"review","letters_reversed":"accept","letters_shuffle":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7,"shuffle5":0.8,"letters":0.25,"letters_reversed":0.7,"letters_shuffle":0.8}}}},"usage":{"input_tokens":700,"output_tokens":7,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"uncalibrated","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":null,"escalate":null,"variants":"fast"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":null,"soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"parallel-auto","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel2","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"B","probabilities":{"A":0.25,"B":0.75,"C":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["rename"],"option_layout":"parallel","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":0.5,"reorder_stability":null,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","letters":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"letters":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"parallel-explicit-reorder","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6,"probes":"reorder"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["review: Review the request","accept: Accept the request","reject: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder"],"option_layout":"parallel","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":null,"reorder_stability":0.5,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","shuffle2":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle2":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"parallel-relabel-off","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6,"relabel":false},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":[],"option_layout":"parallel","decision":null,"soft_stability":null,"stability":null,"rename_stability":null,"reorder_stability":null,"n_variants":1,"distinct_choices":["accept"],"variant_choices":{"original":"accept"},"variant_support":{"original":0.8}}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"rename-cap","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"v0":"description 0","v1":"description 1","v2":"description 2","v3":"description 3","v4":"description 4","v5":"description 5","v6":"description 6","v7":"description 7","v8":"description 8","v9":"description 9","v10":"description 10","v11":"description 11","v12":"description 12","v13":"description 13","v14":"description 14","v15":"description 15","v16":"description 16","v17":"description 17","v18":"description 18","v19":"description 19"}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["v0: description 0","v1: description 1","v2: description 2","v3: description 3","v4: description 4","v5: description 5","v6: description 6","v7: description 7","v8: description 8","v9: description 9","v10: description 10","v11: description 11","v12: description 12","v13: description 13","v14: description 14","v15: description 15","v16: description 16","v17: description 17","v18: description 18","v19: description 19"]},{"key":"q::rel1","rendered":["A: description 19","B: description 18","C: description 17","D: description 16","E: description 15","F: description 14","G: description 13","H: description 12","I: description 11","J: description 10","K: description 9","L: description 8","M: description 7","N: description 6","O: description 5","P: description 4","Q: description 3","R: description 2","S: description 1","T: description 0"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0}},"q::rel1":{"type":"choice","choice":"S","probabilities":{"T":0.25,"S":0.75,"R":0.0,"Q":0.0,"P":0.0,"O":0.0,"N":0.0,"M":0.0,"L":0.0,"K":0.0,"J":0.0,"I":0.0,"H":0.0,"G":0.0,"F":0.0,"E":0.0,"D":0.0,"C":0.0,"B":0.0,"A":0.0}}},"usage":{"input_tokens":200,"output_tokens":2,"truncated_questions":["q","q::rel1"],"options":{"q":{"total":20,"distinct":20},"q::rel1":{"total":20,"distinct":20}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0},"reliability":{"probes":["rename"],"option_layout":"parallel","decision":"ESCALATE","soft_stability":0.525,"stability":0.0,"rename_stability":0.0,"reorder_stability":null,"n_variants":2,"distinct_choices":["v0","v1"],"variant_choices":{"original":"v0","letters_reversed":"v1"},"variant_support":{"original":0.8,"letters_reversed":0.25}}}},"usage":{"input_tokens":200,"output_tokens":2,"truncated_questions":["q"],"options":{"q":{"total":20,"distinct":20}}}}]},+ {"name":"over-rename-cap","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"v0":null,"v1":null,"v2":null,"v3":null,"v4":null,"v5":null,"v6":null,"v7":null,"v8":null,"v9":null,"v10":null,"v11":null,"v12":null,"v13":null,"v14":null,"v15":null,"v16":null,"v17":null,"v18":null,"v19":null,"v20":null}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0,"v20":0.0}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":21,"distinct":21}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0,"v20":0.0},"reliability":{"probes":[],"option_layout":"parallel","decision":null,"soft_stability":null,"stability":null,"rename_stability":null,"reorder_stability":null,"n_variants":1,"distinct_choices":["v0"],"variant_choices":{"original":"v0"},"variant_support":{"original":0.8}}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":21,"distinct":21}}}}]},+ {"name":"alphabet-cap","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"v0":null,"v1":null,"v2":null,"v3":null,"v4":null,"v5":null,"v6":null,"v7":null,"v8":null,"v9":null,"v10":null,"v11":null,"v12":null,"v13":null,"v14":null,"v15":null,"v16":null,"v17":null,"v18":null,"v19":null,"v20":null,"v21":null,"v22":null,"v23":null,"v24":null,"v25":null,"v26":null}}},"layout":"parallel","options":{"accept":0.8,"escalate":0.6,"renameMaxOptions":27},"expanded":[{"key":"q","rendered":["v0","v1","v2","v3","v4","v5","v6","v7","v8","v9","v10","v11","v12","v13","v14","v15","v16","v17","v18","v19","v20","v21","v22","v23","v24","v25","v26"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0,"v20":0.0,"v21":0.0,"v22":0.0,"v23":0.0,"v24":0.0,"v25":0.0,"v26":0.0}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":27,"distinct":27}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"v0","probabilities":{"v0":0.8,"v1":0.19999999999999996,"v2":0.0,"v3":0.0,"v4":0.0,"v5":0.0,"v6":0.0,"v7":0.0,"v8":0.0,"v9":0.0,"v10":0.0,"v11":0.0,"v12":0.0,"v13":0.0,"v14":0.0,"v15":0.0,"v16":0.0,"v17":0.0,"v18":0.0,"v19":0.0,"v20":0.0,"v21":0.0,"v22":0.0,"v23":0.0,"v24":0.0,"v25":0.0,"v26":0.0},"reliability":{"probes":[],"option_layout":"parallel","decision":null,"soft_stability":null,"stability":null,"rename_stability":null,"reorder_stability":null,"n_variants":1,"distinct_choices":["v0"],"variant_choices":{"original":"v0"},"variant_support":{"original":0.8}}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":27,"distinct":27}}}}]},+ {"name":"one-option","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"only":null}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6},"expanded":[{"key":"q","rendered":["only"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"only","probabilities":{"only":1.0}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":1,"distinct":1}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"only","probabilities":{"only":1.0},"reliability":{"probes":[],"option_layout":"sequential","decision":null,"soft_stability":null,"stability":null,"rename_stability":null,"reorder_stability":null,"n_variants":1,"distinct_choices":["only"],"variant_choices":{"original":"only"},"variant_support":{"original":1.0}}}},"usage":{"input_tokens":100,"output_tokens":1,"truncated_questions":["q"],"options":{"q":{"total":1,"distinct":1}}}}]},+ {"name":"batch-seed","states":["first","second"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}},{"answers":{"q":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel1":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}},"q::rel2":{"type":"choice","choice":"C","probabilities":{"C":0.8,"B":0.19999999999999996,"A":0.0}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}},{"answers":{"q":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.4166666666666667,"stability":0.0,"rename_stability":0.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["accept","review"],"variant_choices":{"original":"review","reversed":"accept","letters_reversed":"accept"},"variant_support":{"original":0.75,"reversed":0.3,"letters_reversed":0.2}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"explicit-structured-state","states":[{"user":"你好","enabled":true}],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"accept":"Accept the request","review":"Review the request","reject":"Reject the request"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"seed":"shared seed"},"expanded":[{"key":"q","rendered":["accept: Accept the request","review: Review the request","reject: Reject the request"]},{"key":"q::rel1","rendered":["reject: Reject the request","review: Review the request","accept: Accept the request"]},{"key":"q::rel2","rendered":["reject: Reject the request","accept: Accept the request","review: Review the request"]},{"key":"q::rel3","rendered":["accept: Accept the request","reject: Reject the request","review: Review the request"]},{"key":"q::rel4","rendered":["A: Accept the request","B: Review the request","C: Reject the request"]},{"key":"q::rel5","rendered":["A: Reject the request","B: Review the request","C: Accept the request"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel1":{"type":"choice","choice":"review","probabilities":{"accept":0.25,"review":0.75,"reject":0.0}},"q::rel2":{"type":"choice","choice":"accept","probabilities":{"accept":0.7,"review":0.30000000000000004,"reject":0.0}},"q::rel3":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0}},"q::rel4":{"type":"choice","choice":"B","probabilities":{"A":0.25,"B":0.75,"C":0.0}},"q::rel5":{"type":"choice","choice":"C","probabilities":{"C":0.7,"B":0.30000000000000004,"A":0.0}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q","q::rel1","q::rel2","q::rel3","q::rel4","q::rel5"],"options":{"q":{"total":3,"distinct":3},"q::rel1":{"total":3,"distinct":3},"q::rel2":{"total":3,"distinct":3},"q::rel3":{"total":3,"distinct":3},"q::rel4":{"total":3,"distinct":3},"q::rel5":{"total":3,"distinct":3}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"accept","probabilities":{"accept":0.8,"review":0.19999999999999996,"reject":0.0},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5833333333333334,"stability":0.6,"rename_stability":0.5,"reorder_stability":0.6667,"n_variants":6,"distinct_choices":["accept","review"],"variant_choices":{"original":"accept","reversed":"review","shuffle1":"accept","shuffle2":"accept","letters":"review","letters_reversed":"accept"},"variant_support":{"original":0.8,"reversed":0.25,"shuffle1":0.7,"shuffle2":0.8,"letters":0.25,"letters_reversed":0.7}}}},"usage":{"input_tokens":600,"output_tokens":6,"truncated_questions":["q"],"options":{"q":{"total":3,"distinct":3}}}}]},+ {"name":"rounding-exact-halves","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.9,"no":0.09999999999999998}},"q::rel1":{"type":"choice","choice":"no","probabilities":{"yes":0.03125,"no":0.96875}},"q::rel2":{"type":"choice","choice":"A","probabilities":{"B":0.15625,"A":0.84375}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.9,"no":0.09999999999999998},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.3625,"stability":0.0,"rename_stability":0.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["no","yes"],"variant_choices":{"original":"yes","reversed":"no","letters_reversed":"no"},"variant_support":{"original":0.9,"reversed":0.0312,"letters_reversed":0.1562}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},+ {"name":"rounding-binary-neighbors","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.9,"no":0.09999999999999998}},"q::rel1":{"type":"choice","choice":"no","probabilities":{"yes":0.12345,"no":0.8765499999999999}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.99995,"A":4.999999999999449e-05}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.9,"no":0.09999999999999998},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.6744666666666667,"stability":0.5,"rename_stability":1.0,"reorder_stability":0.0,"n_variants":3,"distinct_choices":["no","yes"],"variant_choices":{"original":"yes","reversed":"no","letters_reversed":"yes"},"variant_support":{"original":0.9,"reversed":0.1235,"letters_reversed":1.0}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},+ {"name":"below-accept","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.7999,"no":0.20009999999999994}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.8,"A":0.19999999999999996}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.7999,"no":0.20009999999999994},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.7999666666666667,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.7999,"reversed":0.8,"letters_reversed":0.8}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},+ {"name":"at-accept","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.8,"A":0.19999999999999996}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ACCEPT","soft_stability":0.8000000000000002,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.8,"reversed":0.8,"letters_reversed":0.8}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},+ {"name":"above-accept","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.8001,"no":0.19989999999999997}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.8,"no":0.19999999999999996}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.8,"A":0.19999999999999996}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.8001,"no":0.19989999999999997},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ACCEPT","soft_stability":0.8000333333333334,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.8001,"reversed":0.8,"letters_reversed":0.8}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},+ {"name":"below-escalate","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.5999,"no":0.4001}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.6,"no":0.4}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.6,"A":0.4}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.5999,"no":0.4001},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"ESCALATE","soft_stability":0.5999666666666666,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.5999,"reversed":0.6,"letters_reversed":0.6}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]},+ {"name":"at-escalate","states":["fixture state"],"questions":{"q":{"type":"choice","instructions":"Choose a suitable action.","criteria":{"yes":"allow","no":"deny"}}},"layout":"sequential","options":{"accept":0.8,"escalate":0.6,"variants":"fast"},"expanded":[{"key":"q","rendered":["yes: allow","no: deny"]},{"key":"q::rel1","rendered":["no: deny","yes: allow"]},{"key":"q::rel2","rendered":["A: deny","B: allow"]}],"model_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.6,"no":0.4}},"q::rel1":{"type":"choice","choice":"yes","probabilities":{"yes":0.6,"no":0.4}},"q::rel2":{"type":"choice","choice":"B","probabilities":{"B":0.6,"A":0.4}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q","q::rel1","q::rel2"],"options":{"q":{"total":2,"distinct":2},"q::rel1":{"total":2,"distinct":2},"q::rel2":{"total":2,"distinct":2}}}}],"expected_results":[{"answers":{"q":{"type":"choice","choice":"yes","probabilities":{"yes":0.6,"no":0.4},"reliability":{"probes":["reorder","rename"],"option_layout":"sequential","decision":"VERIFY","soft_stability":0.6,"stability":1.0,"rename_stability":1.0,"reorder_stability":1.0,"n_variants":3,"distinct_choices":["yes"],"variant_choices":{"original":"yes","reversed":"yes","letters_reversed":"yes"},"variant_support":{"original":0.6,"reversed":0.6,"letters_reversed":0.6}}}},"usage":{"input_tokens":300,"output_tokens":3,"truncated_questions":["q"],"options":{"q":{"total":2,"distinct":2}}}}]}+ ],+ "calibration_cases": [+ {"name":"near-equal-cuts","scores":[0.79996,0.8,0.80004],"correct":[false,false,true],"options":{"targetAccuracy":1.0},"thresholds":[0.80004,0.8],"decisions":["ESCALATE","VERIFY","ACCEPT"]},+ {"name":"mixed-neighbors","scores":[0.59996,0.6,0.60004,0.79996,0.8,0.80004],"correct":[false,true,false,true,true,true],"options":{},"thresholds":[0.79996,0.6],"decisions":["ESCALATE","VERIFY","VERIFY","ACCEPT","ACCEPT","ACCEPT"]},+ {"name":"equal-score-cohort","scores":[0.8,0.8,0.9],"correct":[true,false,true],"options":{"targetAccuracy":1.0},"thresholds":[0.9,0.0],"decisions":["VERIFY","VERIFY","ACCEPT"]},+ {"name":"accept-nothing","scores":[0.2,0.8,0.9],"correct":[false,false,false],"options":{},"thresholds":[1.01,0.9],"decisions":["ESCALATE","ESCALATE","VERIFY"]},+ {"name":"all-correct","scores":[0.2,0.8,0.9],"correct":[true,true,true],"options":{},"thresholds":[0.2,0.0],"decisions":["ACCEPT","ACCEPT","ACCEPT"]},+ {"name":"edges","scores":[0.0,1.0],"correct":[false,true],"options":{"maxEscalateAccuracy":0.0},"thresholds":[1.0,0.0],"decisions":["VERIFY","ACCEPT"]}+ ],+ "decision_cases": [+ {"score":0.0,"accept":null,"escalate":null,"expected":null},+ {"score":0.5999999999999999,"accept":null,"escalate":null,"expected":null},+ {"score":0.6,"accept":null,"escalate":null,"expected":null},+ {"score":0.7999999999999999,"accept":null,"escalate":null,"expected":null},+ {"score":0.8,"accept":null,"escalate":null,"expected":null},+ {"score":0.8000000000000002,"accept":null,"escalate":null,"expected":null},+ {"score":1.0,"accept":null,"escalate":null,"expected":null},+ {"score":0.0,"accept":0.8,"escalate":0.6,"expected":"ESCALATE"},+ {"score":0.5999999999999999,"accept":0.8,"escalate":0.6,"expected":"ESCALATE"},+ {"score":0.6,"accept":0.8,"escalate":0.6,"expected":"VERIFY"},+ {"score":0.7999999999999999,"accept":0.8,"escalate":0.6,"expected":"VERIFY"},+ {"score":0.8,"accept":0.8,"escalate":0.6,"expected":"ACCEPT"},+ {"score":0.8000000000000002,"accept":0.8,"escalate":0.6,"expected":"ACCEPT"},+ {"score":1.0,"accept":0.8,"escalate":0.6,"expected":"ACCEPT"},+ {"score":0.0,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},+ {"score":0.5999999999999999,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},+ {"score":0.6,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},+ {"score":0.7999999999999999,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},+ {"score":0.8,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},+ {"score":0.8000000000000002,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},+ {"score":1.0,"accept":1.01,"escalate":0.0,"expected":"VERIFY"},+ {"score":0.0,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},+ {"score":0.5999999999999999,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},+ {"score":0.6,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},+ {"score":0.7999999999999999,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},+ {"score":0.8,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},+ {"score":0.8000000000000002,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"},+ {"score":1.0,"accept":0.0,"escalate":0.0,"expected":"ACCEPT"}+ ],+ "pipeline": {+ "cases": [+ "below-accept",+ "at-accept",+ "above-accept"+ ],+ "correct": [+ false,+ false,+ true+ ],+ "options": {+ "targetAccuracy": 1.0+ },+ "thresholds": [+ 0.8000333333333334,+ 0.8000000000000002+ ],+ "decisions": [+ "ESCALATE",+ "VERIFY",+ "ACCEPT"+ ]+ }+}diff --git a/laya-ts/scripts/option_stability_fixtures.py b/laya-ts/scripts/option_stability_fixtures.pynew file mode 100644index 00000000..004c2169--- /dev/null+++ b/laya-ts/scripts/option_stability_fixtures.py@@ -0,0 +1,276 @@+"""Generate model-free TypeScript fixtures from production Python stability hooks.++ python laya-ts/scripts/option_stability_fixtures.py --python-root REFERENCE+ python laya-ts/scripts/option_stability_fixtures.py --python-root REFERENCE --check++REFERENCE must contain laya/hooks.py and laya/common.py. The committed oracle records+its actual source hashes and Git provenance; it uses the rename semantics from #1286+and the raw score/report/calibration contract from #1289. It is a local combination+of those changes until they land, not a claim that they are already on main.+Only the standard library is needed. The production render functions are loaded+unchanged from their AST so generating fixtures does not import torch or a model.+"""+from __future__ import annotations++import argparse+import ast+from copy import deepcopy+import hashlib+import importlib.util+import json+import math+from pathlib import Path+import platform+import subprocess+import sys+from types import SimpleNamespace+++ROOT = Path(__file__).resolve().parents[2]+FIXTURE = ROOT / "laya-ts/fixtures/option-stability.json"+OPTION_NAMES = {+ "nShuffles": "n_shuffles", "renameMaxOptions": "rename_max_options",+ "targetAccuracy": "target_accuracy", "maxEscalateAccuracy": "max_escalate_accuracy",+}+++def python_options(options):+ return {OPTION_NAMES.get(key, key): value for key, value in options.items()}+++def load_reference(root):+ path = root / "laya/hooks.py"+ spec = importlib.util.spec_from_file_location("stability_fixture_reference", path)+ hooks = importlib.util.module_from_spec(spec)+ sys.modules[spec.name] = hooks+ spec.loader.exec_module(hooks)+ render_names = {"render_criterion", "_resolve_noul_labels", "render_options"}+ tree = ast.parse((root / "laya/common.py").read_text(encoding="utf-8"))+ functions = [node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name in render_names]+ if {node.name for node in functions} != render_names:+ raise RuntimeError("reference common.py is missing the production render helpers")+ scope = {"json": json, "Dict": dict, "List": list,+ "_DEFAULT_NOUL_LABELS": {"false": "false", "true": "true"}}+ exec(compile(ast.Module(body=functions, type_ignores=[]), str(root / "laya/common.py"), "exec"), scope)+ return hooks, scope["render_options"]+++def provenance(root):+ hashes = {name: hashlib.sha256((root / name).read_bytes()).hexdigest()+ for name in ("laya/hooks.py", "laya/common.py")}+ manifest = root / "reference-provenance.json"+ if manifest.exists():+ source = json.loads(manifest.read_text(encoding="utf-8"))+ if source.get("files") != hashes:+ raise RuntimeError("reference files differ from their provenance manifest")+ else:+ def git(*args):+ return subprocess.check_output(["git", "-C", str(root), *args], text=True).strip()+ source = {"commit": git("rev-parse", "HEAD"),+ "modified_reference_files": git("status", "--porcelain", "--", "laya/hooks.py", "laya/common.py"),+ "files": hashes}+ return {**source, "python": platform.python_version()}+++def variants(hooks):+ cases = [+ ("two-dedup", ["yes", "no"], "", {}),+ ("three-utf8", ["alpha", "beta", "gamma"], "种子|café|🙂", {}),+ ("twenty-zero-shuffles", [f"v{i}" for i in range(20)], "twenty", {"nShuffles": 0}),+ # Enough draws to cross MT19937's 624-word state boundary, not only its first block.+ ("twenty-six-twenty-shuffles", [f"v{i}" for i in range(26)], "twenty-six", {"nShuffles": 20}),+ ("twenty-seven-no-letters", [f"v{i}" for i in range(27)], "twenty-seven", {}),+ ("fast-include", ["a", "b", "c"], "seed", {"include": ["reversed", "letters_reversed"]}),+ ("custom-include", ["a", "b", "c", "d"], "seed", {"include": ["shuffle3", "letters_shuffle"]}),+ ("rename-only", ["a", "b", "c"], "seed", {"reorder": False}),+ ("reorder-only", ["a", "b", "c"], "seed", {"relabel": False}),+ ("no-probes", ["a", "b", "c"], "seed", {"relabel": False, "reorder": False}),+ ("filtered-away", ["a", "b", "c"], "seed", {"nShuffles": 0, "include": ["shuffle3"]}),+ ("unicode-labels", ["\ue000", "🙂", "é"], "nul\u0000seed", {"nShuffles": 5}),+ ]+ return [{"name": name, "labels": labels, "seed_key": seed, "options": options,+ "expected": [{"name": n, "order": order, "relabel": rel}+ for n, order, rel in hooks.make_variants(labels, seed, **python_options(options))]}+ for name, labels, seed, options in cases]+++def hook_case(hooks, render_options, name, criteria, *, options=None, layout="sequential",+ order=None, states=None, supports=None):+ options = {"accept": 0.8, "escalate": 0.6, **(options or {})}+ q = {"type": "choice", "instructions": "Choose a suitable action.", "criteria": criteria}+ if order is not None:+ q["option_order"] = order+ questions = {"q": q}+ original = deepcopy(questions)+ states = ["fixture state"] if states is None else states+ hook = hooks.OptionStabilityHook(**python_options(options))+ ctx = hooks.PredictContext(states=deepcopy(states), questions=questions,+ agent=SimpleNamespace(config={"option_layout": layout}))+ hook.on_predict_start(ctx)+ assert questions == original, "the reference mutated caller-owned questions"++ expanded = []+ for key, question in ctx.questions.items():+ crit = question["criteria"]+ canonical = dict.fromkeys(crit) if isinstance(crit, list) else crit+ rendered = render_options({"t": "choice", "crit": canonical})+ physical = question.get("option_order", list(range(len(rendered))))+ expanded.append({"key": key, "rendered": [rendered[i] for i in physical]})++ # The reference's map is only used to synthesize model rows in each probe's+ # displayed label space. Expected public results still come from on_predict_end.+ plan = hook._plans[ctx.run_id]["q"]["variants"]+ labels = list(criteria)+ target = labels[0]+ others = labels[1:]+ model_results = []+ for state_index in range(len(states)):+ answers = {}+ for i, (_, key, back, _) in enumerate(plan):+ support = supports[i % len(supports)] if supports else (0.8, 0.25, 0.7)[(i + state_index) % 3]+ if not others:+ support = 1.0+ chosen = target if support >= 0.5 or not others else others[0]+ forward = {original: shown for shown, original in back.items()}+ probabilities = {forward[label]: (support if label == target else+ 1.0 - support if label == others[0] else 0.0) for label in labels}+ answers[key] = {"type": "choice", "choice": forward[chosen], "probabilities": probabilities}+ usage = {"input_tokens": 100 * len(answers), "output_tokens": len(answers),+ "truncated_questions": list(answers),+ "options": {key: {"total": len(labels), "distinct": len(labels)} for key in answers}}+ model_results.append({"answers": answers, "usage": usage})+ ctx.results = deepcopy(model_results)+ hook.on_predict_end(ctx)+ assert questions == original, "end hook mutated caller-owned questions"+ assert ctx.run_id not in hook._plans, "reference leaked its completed plan"+ for result in ctx.results:+ report = result["answers"]["q"]["reliability"]+ if report["soft_stability"] is not None:+ assert hooks.decide_reliability(report["soft_stability"], options["accept"], options["escalate"]) == report["decision"]+ return {"name": name, "states": states, "questions": original, "layout": layout,+ "options": options, "expanded": expanded,+ "model_results": model_results, "expected_results": ctx.results}+++def hook_cases(hooks, render_options):+ make = lambda name, criteria, **kw: hook_case(hooks, render_options, name, criteria, **kw)+ described = {"accept": "Accept the request", "review": "Review the request", "reject": "Reject the request"}+ cases = [+ make("described-full", described),+ make("list-fast", list(described), options={"variants": "fast"}),+ make("undescribed", {"accept": None, "review": "", "reject": " \t\n"}, options={"variants": "fast"}),+ make("mixed-values", {"missing": None, "blank": " ", "zero": 0, "false": False,+ "object": {"detail": "café", "priority": 2}, "array": ["x", 3]},+ options={"variants": "fast"}),+ make("numeric-keys-order", {"1": "first", "2": "second", "10": "tenth"}, order=[2, 0, 1]),+ make("numeric-list", ["1", "2", "10"], options={"variants": "fast"}),+ make("prototype-labels", {"__proto__": "base", "constructor": "create", "toString": "print"}),+ make("unicode-labels", {"\ue000": "private", "🙂": "smile", "é": "accent"}),+ make("custom-variants", described, options={"variants": ["shuffle3", "letters_shuffle"], "seed": "种子🙂"}),+ make("zero-shuffles", described, options={"nShuffles": 0}),+ make("extra-shuffles", described, options={"nShuffles": 5}),+ make("uncalibrated", described, options={"accept": None, "escalate": None, "variants": "fast"}),+ make("parallel-auto", described, layout="parallel"),+ make("parallel-explicit-reorder", described, layout="parallel", options={"probes": "reorder"}),+ make("parallel-relabel-off", described, layout="parallel", options={"relabel": False}),+ make("rename-cap", {f"v{i}": f"description {i}" for i in range(20)}, layout="parallel", options={"variants": "fast"}),+ make("over-rename-cap", {f"v{i}": None for i in range(21)}, layout="parallel"),+ make("alphabet-cap", {f"v{i}": None for i in range(27)}, layout="parallel", options={"renameMaxOptions": 27}),+ make("one-option", {"only": None}),+ make("batch-seed", described, states=["first", "second"], options={"variants": "fast"}),+ make("explicit-structured-state", described, states=[{"user": "你好", "enabled": True}], options={"seed": "shared seed"}),+ make("rounding-exact-halves", {"yes": "allow", "no": "deny"},+ options={"variants": "fast"}, supports=[0.9, 0.03125, 0.15625]),+ make("rounding-binary-neighbors", {"yes": "allow", "no": "deny"},+ options={"variants": "fast"}, supports=[0.9, 0.12345, 0.99995]),+ ]+ for name, support in [("below-accept", [0.7999, 0.8, 0.8]),+ ("at-accept", [0.8, 0.8, 0.8]),+ ("above-accept", [0.8001, 0.8, 0.8]),+ ("below-escalate", [0.5999, 0.6, 0.6]),+ ("at-escalate", [0.6, 0.6, 0.6])]:+ case = make(name, {"yes": "allow", "no": "deny"}, options={"variants": "fast"}, supports=support)+ report = case["expected_results"][0]["answers"]["q"]["reliability"]+ assert report["soft_stability"] == sum(support) / 3, "reference needs #1289 raw scores"+ cases.append(case)+ renamed = next(case for case in cases if case["name"] == "undescribed")["expanded"][-1]["rendered"]+ assert renamed == ["A: reject", "B: review", "C: accept"], "reference needs #1286 label semantics"+ return cases+++def calibrations(hooks):+ cases = [+ ("near-equal-cuts", [0.79996, 0.8, 0.80004], [False, False, True], {"targetAccuracy": 1.0}),+ ("mixed-neighbors", [0.59996, 0.6, 0.60004, 0.79996, 0.8, 0.80004], [False, True, False, True, True, True], {}),+ ("equal-score-cohort", [0.8, 0.8, 0.9], [True, False, True], {"targetAccuracy": 1.0}),+ ("accept-nothing", [0.2, 0.8, 0.9], [False, False, False], {}),+ ("all-correct", [0.2, 0.8, 0.9], [True, True, True], {}),+ ("edges", [0.0, 1.0], [False, True], {"maxEscalateAccuracy": 0.0}),+ ]+ result = []+ for name, scores, correct, options in cases:+ thresholds = hooks.calibrate_reliability(scores, correct, **python_options(options))+ result.append({"name": name, "scores": scores, "correct": correct, "options": options,+ "thresholds": thresholds,+ "decisions": [hooks.decide_reliability(score, *thresholds) for score in scores]})+ assert result[0]["thresholds"][0] == 0.80004, "reference needs #1289 distinct raw cuts"+ return result+++def generate(root):+ hooks, render_options = load_reference(root)+ cases = hook_cases(hooks, render_options)+ decisions = [{"score": score, "accept": accept, "escalate": escalate,+ "expected": hooks.decide_reliability(score, accept, escalate)}+ for accept, escalate in [(None, None), (0.8, 0.6), (1.01, 0.0), (0.0, 0.0)]+ for score in [0.0, math.nextafter(0.6, 0.0), 0.6, math.nextafter(0.8, 0.0), 0.8,+ math.nextafter(0.8, 1.0), 1.0]]+ selected = [case for case in cases if case["name"] in ("below-accept", "at-accept", "above-accept")]+ scores = [case["expected_results"][0]["answers"]["q"]["reliability"]["soft_stability"] for case in selected]+ correct = [False, False, True]+ thresholds = hooks.calibrate_reliability(scores, correct, target_accuracy=1.0)+ return {"schema": "laya-option-stability/1", "reference": provenance(root),+ "variant_cases": variants(hooks), "hook_cases": cases,+ "calibration_cases": calibrations(hooks), "decision_cases": decisions,+ "pipeline": {"cases": [case["name"] for case in selected], "correct": correct,+ "options": {"targetAccuracy": 1.0}, "thresholds": thresholds,+ "decisions": [hooks.decide_reliability(score, *thresholds) for score in scores]}}+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--python-root", type=Path, default=ROOT)+ parser.add_argument("--fixture", type=Path, default=FIXTURE)+ parser.add_argument("--check", action="store_true", help="compare behavior with the committed oracle without writing")+ args = parser.parse_args()+ result = generate(args.python_root.resolve())+ if args.check:+ expected = json.loads(args.fixture.read_text(encoding="utf-8"))+ # A newer source commit may implement the same contract. Compare behavior;+ # print actual provenance, retaining the original oracle's provenance.+ actual = {key: value for key, value in result.items() if key != "reference"}+ wanted = {key: value for key, value in expected.items() if key != "reference"}+ if json.loads(json.dumps(actual)) != wanted:+ raise SystemExit("option stability fixtures differ from this Python reference")+ print("PASS: Python reference matches all committed option stability fixtures")+ else:+ args.fixture.parent.mkdir(parents=True, exist_ok=True)+ # Keep one complete case per line: the oracle stays small while its named+ # cases and source provenance remain easy to locate and inspect.+ fields = []+ for key, value in result.items():+ if key.endswith("_cases"):+ encoded = "[\n" + ",\n".join(" " + json.dumps(case, ensure_ascii=False, separators=(",", ":"))+ for case in value) + "\n ]"+ else:+ encoded = json.dumps(value, ensure_ascii=False, indent=2).replace("\n", "\n ")+ fields.append(f" {json.dumps(key)}: {encoded}")+ args.fixture.write_text("{\n" + ",\n".join(fields) + "\n}\n", encoding="utf-8")+ print(f"Wrote {args.fixture} ({args.fixture.stat().st_size} bytes)")+ print(json.dumps(result["reference"], ensure_ascii=False, sort_keys=True))+ print(f"Cases: {len(result['variant_cases'])} variants, {len(result['hook_cases'])} hook calls, "+ f"{len(result['calibration_cases'])} calibrations, {len(result['decision_cases'])} decisions, 1 fit/replay pipeline")+++if __name__ == "__main__":+ main()diff --git a/laya-ts/src/_option-stability-random.ts b/laya-ts/src/_option-stability-random.tsnew file mode 100644index 00000000..c87a7ea6--- /dev/null+++ b/laya-ts/src/_option-stability-random.ts@@ -0,0 +1,65 @@+/** Python's integer-seeded random.Random, restricted to the shuffle operation used by+ * option stability. Keep it private: changing the RNG changes which probes run. */+export class StabilityRandom {+ private readonly state = new Uint32Array(624);+ private index = 624;++ constructor(seedKey: string) {+ let crc = 0xffffffff;+ for (const byte of new TextEncoder().encode(seedKey)) {+ crc ^= byte;+ for (let bit = 0; bit < 8; bit++) crc = (crc >>> 1) ^ ((crc & 1) ? 0xedb88320 : 0);+ }+ const seed = (crc ^ 0xffffffff) >>> 0;+ // CPython passes the integer's little-endian words to MT19937 init_by_array.+ // CRC32 fits in one word; init_genrand(seed) alone produces different shuffles.+ const mt = this.state;+ mt[0] = 19650218;+ for (let i = 1; i < mt.length; i++) {+ mt[i] = (Math.imul(mt[i - 1] ^ (mt[i - 1] >>> 30), 1812433253) + i) >>> 0;+ }+ let i = 1;+ for (let k = 624; k > 0; k--) {+ mt[i] = ((mt[i] ^ Math.imul(mt[i - 1] ^ (mt[i - 1] >>> 30), 1664525)) + seed) >>> 0;+ if (++i >= 624) { mt[0] = mt[623]; i = 1; }+ }+ for (let k = 623; k > 0; k--) {+ mt[i] = ((mt[i] ^ Math.imul(mt[i - 1] ^ (mt[i - 1] >>> 30), 1566083941)) - i) >>> 0;+ if (++i >= 624) { mt[0] = mt[623]; i = 1; }+ }+ mt[0] = 0x80000000;+ }++ private next(): number {+ const mt = this.state;+ if (this.index >= 624) {+ for (let i = 0; i < 624; i++) {+ const y = (mt[i] & 0x80000000) | (mt[(i + 1) % 624] & 0x7fffffff);+ mt[i] = mt[(i + 397) % 624] ^ (y >>> 1) ^ ((y & 1) ? 0x9908b0df : 0);+ }+ this.index = 0;+ }+ let y = mt[this.index++];+ y ^= y >>> 11;+ y ^= (y << 7) & 0x9d2c5680;+ y ^= (y << 15) & 0xefc60000;+ y ^= y >>> 18;+ return y >>> 0;+ }++ private below(n: number): number {+ const bits = 32 - Math.clz32(n);+ let r: number;+ // random._randbelow uses n.bit_length(), including when n is a power of two,+ // and consumes another word on rejection. A modulo would bias and desync it.+ do { r = this.next() >>> (32 - bits); } while (r >= n);+ return r;+ }++ shuffle<T>(values: T[]): void {+ for (let i = values.length - 1; i > 0; i--) {+ const j = this.below(i + 1);+ [values[i], values[j]] = [values[j], values[i]];+ }+ }+}diff --git a/laya-ts/src/agent.ts b/laya-ts/src/agent.tsindex 1a7b4ec6..e6d96659 100644--- a/laya-ts/src/agent.ts+++ b/laya-ts/src/agent.ts@@ -21,6 +21,7 @@ import {} from "./common.js";import type { BinningMap, InternalQ, MinConfidence, OptionStats, SequenceStats } from "./common.js";import type { Batch, SessionProvider } from "./providers.js";+import type { OptionStabilityReport } from "./option-stability.js";import { encodeWithData, parseTokenizerJson, type TokenizerLike } from "./tokenizer.js";import { decide, decideBatch, type DecideOptions, type DecideBatchOptions, type DecisionResult } from "./structured.js";import {@@ -56,6 +57,7 @@ export interface ChoiceAnswer {answer_confidence: number;action: ActionInfo;low_confidence?: boolean;+ reliability?: OptionStabilityReport;}export interface ScoreAnswer {diff --git a/laya-ts/src/index.ts b/laya-ts/src/index.tsindex 21c9f921..cec8a0d0 100644--- a/laya-ts/src/index.ts+++ b/laya-ts/src/index.ts@@ -37,6 +37,11 @@ export {HOOK_EVENTS,} from "./hooks.js";export type { Hook, HookEvent, HookArg, PredictHook, PredictHookArg } from "./hooks.js";+export { OptionStabilityHook, makeVariants, decideReliability, calibrateReliability } from "./option-stability.js";+export type {+ OptionStabilityOptions, OptionStabilityReport, StabilityProbe, ReliabilityDecision,+ StabilityVariant, VariantOptions, CalibrationOptions,+} from "./option-stability.js";export { triageQuestions, emailQuestions, guardQuestions, moderationQuestions, routerQuestions } from "./presets.js";export {decide,diff --git a/laya-ts/src/option-stability.ts b/laya-ts/src/option-stability.tsnew file mode 100644index 00000000..2060fd3e--- /dev/null+++ b/laya-ts/src/option-stability.ts@@ -0,0 +1,402 @@+import { serializeState } from "./common.js";+import { BaseHook, type PredictContext } from "./hooks.js";+import { StabilityRandom } from "./_option-stability-random.js";++export type StabilityProbe = "reorder" | "rename";+export type ReliabilityDecision = "ACCEPT" | "VERIFY" | "ESCALATE";++export interface StabilityVariant {+ name: string;+ order: string[];+ relabel: boolean;+}++export interface VariantOptions {+ nShuffles?: number;+ relabel?: boolean;+ include?: readonly string[] | null;+ reorder?: boolean;+}++export interface CalibrationOptions {+ targetAccuracy?: number;+ maxEscalateAccuracy?: number;+}++export interface OptionStabilityOptions {+ /** Domain-calibrated thresholds; pass null for uncalibrated metrics only. */+ accept: number | null;+ escalate: number | null;+ probes?: "auto" | "all" | StabilityProbe | readonly StabilityProbe[] | null;+ nShuffles?: number;+ relabel?: boolean;+ renameMaxOptions?: number;+ variants?: "full" | "fast" | readonly string[] | null;+ /** Use an explicit seed for cross-language parity with structured states. */+ seed?: string | null;+}++export interface OptionStabilityReport {+ probes: StabilityProbe[];+ option_layout: "parallel" | "sequential";+ decision: ReliabilityDecision | null;+ /** The unrounded mean used by decideReliability and calibrateReliability. */+ soft_stability: number | null;+ stability: number | null;+ rename_stability: number | null;+ reorder_stability: number | null;+ n_variants: number;+ distinct_choices: string[];+ variant_choices: Record<string, string | null>;+ variant_support: Record<string, number>;+}++const SEP = "::rel";+const ALL_VARIANTS = ["reversed", "shuffle1", "shuffle2", "shuffle3", "letters", "letters_reversed", "letters_shuffle"];++/** Deterministic option permutations, including the original, with duplicates removed.+ * The CRC32 / Python MT19937 sequence matches laya.hooks.make_variants. Filtering+ * happens after shuffling so a named subset retains the full preset's permutations. */+export function makeVariants(labels: readonly string[], seedKey: string, opts: VariantOptions = {}): StabilityVariant[] {+ const { nShuffles = 3, relabel = true, include = null, reorder = true } = opts;+ checkShuffles(nShuffles);+ const rng = new StabilityRandom(seedKey);+ const candidates: StabilityVariant[] = [{ name: "original", order: [...labels], relabel: false }];+ if (reorder) {+ candidates.push({ name: "reversed", order: [...labels].reverse(), relabel: false });+ for (let i = 0; i < nShuffles; i++) {+ const order = [...labels];+ rng.shuffle(order);+ candidates.push({ name: `shuffle${i + 1}`, order, relabel: false });+ }+ }+ if (relabel && labels.length <= 26) {+ candidates.push({ name: "letters", order: [...labels], relabel: true });+ candidates.push({ name: "letters_reversed", order: [...labels].reverse(), relabel: true });+ const order = [...labels];+ rng.shuffle(order);+ candidates.push({ name: "letters_shuffle", order, relabel: true });+ }+ const wanted = include === null ? null : new Set(include);+ const seen = new Set<string>();+ return candidates.filter((v) => {+ if (wanted && v.name !== "original" && !wanted.has(v.name)) return false;+ const key = JSON.stringify([v.order, v.relabel]);+ if (seen.has(key)) return false;+ seen.add(key);+ return true;+ });+}++/** Uncalibrated thresholds fail closed; equality with accept accepts, while equality+ * with escalate verifies. A fitted accept of 1.01 accepts no probability score. */+export function decideReliability(softStability: number, accept: number | null,+ escalate: number | null = null): ReliabilityDecision | null {+ checkProbability(softStability, "softStability");+ checkThresholds(accept, escalate);+ if (accept === null) return null;+ if (softStability >= accept) return "ACCEPT";+ if (escalate !== null && softStability < escalate) return "ESCALATE";+ return "VERIFY";+}++/** Fit [accept, escalate] on labelled examples. Distinct raw scores are candidate+ * cutoffs; equal scores stay together and report JSON can be replayed without rounding.+ * Inputs must be equally sized, non-empty score/label arrays. */+export function calibrateReliability(scores: readonly number[], correct: readonly boolean[],+ opts: CalibrationOptions = {}): [number, number] {+ const { targetAccuracy = 0.90, maxEscalateAccuracy = 1 / 3 } = opts;+ if (scores.length !== correct.length) throw new Error("scores and correct must have the same length");+ if (!scores.length) throw new Error("cannot calibrate reliability on empty scores/correct pairs");+ checkProbability(targetAccuracy, "targetAccuracy");+ checkProbability(maxEscalateAccuracy, "maxEscalateAccuracy");+ const pairs = Array.from(scores, (score, i) => {+ checkProbability(score, `scores[${i}]`);+ if (typeof correct[i] !== "boolean") throw new TypeError(`correct[${i}] must be a bool`);+ return [score, correct[i]] as const;+ });+ const cuts = [...new Set(pairs.map(([score]) => score))].sort((a, b) => a - b);+ let accept = 1.01;+ for (const t of cuts) {+ const kept = pairs.filter(([s]) => s >= t);+ if (kept.length && kept.filter(([, c]) => c).length / kept.length >= targetAccuracy) {+ accept = t;+ break;+ }+ }+ let escalate = 0;+ for (const t of cuts) {+ if (t >= accept) break;+ const below = pairs.filter(([s]) => s < t);+ if (below.length && below.filter(([, c]) => c).length / below.length <= maxEscalateAccuracy) escalate = t;+ }+ return [accept, escalate];+}++function isRecord(value: unknown): value is Record<string, unknown> {+ return value !== null && typeof value === "object" && !Array.isArray(value);+}++function checkShuffles(value: number): void {+ if (typeof value !== "number" || !Number.isInteger(value) || value < 0) {+ throw new Error("nShuffles must be a non-negative integer");+ }+}++function checkThreshold(value: number | null, name: string, max: number): void {+ if (value !== null && (typeof value !== "number" || !Number.isFinite(value) || value < 0 || value > max)) {+ throw new Error(`${name} must be null or a number in [0, ${max}]`);+ }+}++function checkProbability(value: number, name: string): void {+ if (typeof value !== "number" || !Number.isFinite(value) || value < 0 || value > 1) {+ throw new Error(`${name} must be a finite number in [0, 1]`);+ }+}++function checkThresholds(accept: number | null, escalate: number | null): void {+ checkThreshold(accept, "accept", 1.01);+ checkThreshold(escalate, "escalate", 1);+ if (accept !== null && escalate !== null && escalate > accept) {+ throw new Error("escalate must not be greater than accept");+ }+}++function resolveProbes(probes: OptionStabilityOptions["probes"]): StabilityProbe[] | null {+ if (probes === undefined || probes === null || probes === "auto") return null;+ if (probes === "all") return ["reorder", "rename"];+ if (probes === "rename" || probes === "reorder") return [probes];+ if (!Array.isArray(probes) || !probes.length || probes.some((p) => p !== "reorder" && p !== "rename")) {+ throw new Error("probes must be 'auto', 'all', 'reorder', 'rename', or a non-empty sequence of probe names");+ }+ return [...new Set(probes)];+}++function resolveVariants(variants: OptionStabilityOptions["variants"]): string[] | null {+ if (variants === undefined || variants === null || variants === "full") return null;+ if (variants === "fast") return ["reversed", "letters_reversed"];+ if (!Array.isArray(variants) || !variants.length || variants.some((v) => !ALL_VARIANTS.includes(v))) {+ throw new Error(`variants must be 'full', 'fast', or a non-empty sequence from ${ALL_VARIANTS.join(", ")}`);+ }+ return [...variants];+}++function detectLayout(ctx: PredictContext): "parallel" | "sequential" {+ const cfg = isRecord(ctx.agent) && isRecord(ctx.agent.cfg) ? ctx.agent.cfg : null;+ const raw = cfg?.option_layout;+ if (typeof raw === "string") {+ const layout = raw.trim().toLowerCase();+ if (layout === "parallel" || layout === "sequential") return layout;+ console.warn(`laya: unknown option_layout ${JSON.stringify(raw)} in agent config; defaulting to 'sequential'`);+ }+ return "sequential";+}++function variantQuestion(q: Record<string, unknown>, order: string[], relabel: boolean):+ { question: Record<string, unknown>; back: Map<string, string> } {+ const shown = relabel ? order.map((_, i) => String.fromCharCode(65 + i)) : [...order];+ const criteria = q.criteria;+ const entries = shown.map((label, i) => {+ const original = order[i];+ let description = Array.isArray(criteria) ? null : (criteria as Record<string, unknown>)[original];+ if (relabel && (description === null || description === undefined ||+ (typeof description === "string" && description.trim() === ""))) description = original;+ return [label, description] as const;+ });+ const nextCriteria = Object.fromEntries(entries);+ const question = { ...q, criteria: Array.isArray(criteria) && !relabel ? [...order] : nextCriteria };+ // Integer-like keys enumerate numerically in JS regardless of insertion order. The+ // encoder already supports a slot permutation, so preserve the requested physical+ // order there, including the caller's permutation of each generated variant.+ const actual = Object.keys(nextCriteria);+ const callerOrder = Array.isArray(q.option_order) ? q.option_order as number[] : order.map((_, i) => i);+ const effectiveOrder = callerOrder.map((i) => actual.indexOf(shown[i]));+ if (Object.hasOwn(q, "option_order") || effectiveOrder.some((i, slot) => i !== slot)) {+ (question as Record<string, unknown>).option_order = effectiveOrder;+ }+ return { question, back: new Map(shown.map((label, i) => [label, order[i]])) };+}++/** Python round(value, 4): round the binary value itself, with ties to even.+ * Multiplying a double by 1e4 first can manufacture a tie, and toFixed ties away. */+function round4(value: number): number {+ if (!Number.isFinite(value) || value === 0) return value;+ const data = new DataView(new ArrayBuffer(8));+ data.setFloat64(0, value);+ const bits = data.getBigUint64(0);+ const exponent = Number((bits >> 52n) & 0x7ffn);+ const mantissa = (bits & ((1n << 52n) - 1n)) | (exponent ? 1n << 52n : 0n);+ const power = exponent ? exponent - 1075 : -1074;+ let numerator = mantissa * 10000n;+ const denominator = power < 0 ? 1n << BigInt(-power) : 1n;+ if (power > 0) numerator <<= BigInt(power);+ let rounded = numerator / denominator;+ const remainder = numerator % denominator;+ if (remainder * 2n > denominator || (remainder * 2n === denominator && (rounded & 1n))) rounded++;+ return (bits >> 63n ? -Number(rounded) : Number(rounded)) / 10000;+}++function compareCodePoints(a: string, b: string): number {+ const x = Array.from(a, (c) => c.codePointAt(0)!);+ const y = Array.from(b, (c) => c.codePointAt(0)!);+ for (let i = 0; i < Math.min(x.length, y.length); i++) if (x[i] !== y[i]) return x[i] - y[i];+ return x.length - y.length;+}++interface PlannedVariant {+ name: string;+ key: string;+ back: Map<string, string>;+ relabel: boolean;+}+interface QuestionPlan {+ layout: "parallel" | "sequential";+ variants: PlannedVariant[];+ probes: StabilityProbe[];+}++/** Opt-in stability probes, batched with the original choice questions. Metrics are+ * signals for calibration, not correctness probabilities. Layout detection selects+ * probes; it does not add parallel-layout support to a provider. */+export class OptionStabilityHook extends BaseHook {+ private readonly accept: number | null;+ private readonly escalate: number | null;+ private readonly probes: StabilityProbe[] | null;+ private readonly nShuffles: number;+ private readonly relabel: boolean;+ private readonly renameMaxOptions: number;+ private readonly include: string[] | null;+ private readonly seed: string | null;+ private readonly plans = new WeakMap<PredictContext, Map<string, QuestionPlan>>();++ constructor(opts: OptionStabilityOptions) {+ super();+ if (!opts || !Object.hasOwn(opts, "accept") || !Object.hasOwn(opts, "escalate")) {+ throw new Error("accept and escalate are required; pass null for uncalibrated metrics");+ }+ checkThresholds(opts.accept, opts.escalate);+ this.accept = opts.accept;+ this.escalate = opts.escalate;+ this.probes = resolveProbes(opts.probes);+ this.nShuffles = opts.nShuffles === undefined ? 3 : opts.nShuffles;+ checkShuffles(this.nShuffles);+ this.relabel = opts.relabel === undefined ? true : opts.relabel;+ if (typeof this.relabel !== "boolean") throw new TypeError("relabel must be a bool");+ this.renameMaxOptions = opts.renameMaxOptions === undefined ? 20 : opts.renameMaxOptions;+ if (!Number.isInteger(this.renameMaxOptions) || this.renameMaxOptions < 2) {+ throw new Error("renameMaxOptions must be an integer >= 2");+ }+ this.include = resolveVariants(opts.variants);+ this.seed = opts.seed === undefined || opts.seed === null ? null : String(opts.seed);+ }++ onPredictStart(ctx: PredictContext): void {+ this.plans.delete(ctx);+ for (const qid of Object.keys(ctx.questions)) {+ if (qid.includes(SEP)) throw new Error(`question id ${JSON.stringify(qid)} must not contain ${JSON.stringify(SEP)}`);+ }+ const layout = detectLayout(ctx);+ const configured = new Set(this.probes ?? (layout === "parallel"+ ? (this.relabel ? ["rename"] : []) : (this.relabel ? ["reorder", "rename"] : ["reorder"])));+ // Python uses repr for structured states; TS keeps its existing serializer.+ // Explicit seed, string states, and multi-state "batch" seeds are shared.+ const baseSeed = this.seed ?? (ctx.states.length === 1 ? serializeState(ctx.states[0]) : "batch");+ const plan = new Map<string, QuestionPlan>();+ const questions = { ...ctx.questions };+ for (const [qid, q] of Object.entries(ctx.questions)) {+ if (!isRecord(q) || q.type !== "choice") continue;+ const criteria = q.criteria;+ if (!Array.isArray(criteria) && !isRecord(criteria)) continue;+ // Match the engine's list-to-dict answer keys, including numeric-like labels.+ const labels = Object.keys(Array.isArray(criteria)+ ? Object.fromEntries(criteria.map((label) => [String(label), null])) : criteria);+ const variants: PlannedVariant[] = [{ name: "original", key: qid,+ back: new Map(labels.map((label) => [label, label])), relabel: false }];+ const probes: StabilityProbe[] = [];+ if (labels.length >= 2) {+ const generated = makeVariants(labels, `${baseSeed}|${qid}`, {+ nShuffles: this.nShuffles,+ relabel: configured.has("rename") && labels.length <= this.renameMaxOptions,+ reorder: configured.has("reorder"), include: this.include,+ }).slice(1);+ if (generated.some((v) => !v.relabel)) probes.push("reorder");+ if (generated.some((v) => v.relabel)) probes.push("rename");+ generated.forEach((variant, i) => {+ const key = `${qid}${SEP}${i + 1}`;+ const { question, back } = variantQuestion(q, variant.order, variant.relabel);+ questions[key] = question;+ variants.push({ name: variant.name, key, back, relabel: variant.relabel });+ });+ }+ plan.set(qid, { layout, variants, probes });+ }+ if (plan.size) {+ ctx.questions = questions;+ this.plans.set(ctx, plan);+ }+ }++ onPredictEnd(ctx: PredictContext): void {+ const plan = this.plans.get(ctx);+ this.plans.delete(ctx);+ if (!plan || !ctx.results) return;+ const internalKeys = new Set([...plan.values()].flatMap((p) => p.variants.slice(1).map((v) => v.key)));+ for (const result of ctx.results) {+ if (!isRecord(result)) continue;+ const answers = result.answers;+ if (isRecord(answers)) {+ for (const [qid, p] of plan) {+ if (!Object.hasOwn(answers, qid)) continue;+ const answer = answers[qid];+ if (!isRecord(answer) || answer.type !== "choice" || typeof answer.choice !== "string") continue;+ const seen: { name: string; choice: string | null; support: number; relabel: boolean }[] = [];+ for (const variant of p.variants) {+ if (!Object.hasOwn(answers, variant.key)) break;+ const value = answers[variant.key];+ if (!isRecord(value) || value.type !== "choice" || typeof value.choice !== "string" ||+ !isRecord(value.probabilities)) break;+ const probabilities = Object.entries(value.probabilities);+ if (probabilities.some(([, v]) => typeof v !== "number" || !Number.isFinite(v) || v < 0 || v > 1)) break;+ const probs = new Map(probabilities.map(([label, probability]) => [variant.back.get(label) ?? label, probability as number]));+ const choice = variant.back.get(value.choice) ?? value.choice;+ seen.push({ name: variant.name, choice, support: probs.get(answer.choice) ?? 0, relabel: variant.relabel });+ }+ // A partial or malformed answer set cannot establish stability.+ if (seen.length !== p.variants.length) continue;+ const others = seen.slice(1);+ const probed = others.length > 0 && p.probes.length > 0;+ const rate = (items: typeof seen): number | null => items.length+ ? round4(items.filter((v) => v.choice === answer.choice).length / items.length) : null;+ const soft = probed ? seen.reduce((sum, v) => sum + v.support, 0) / seen.length : null;+ const report: OptionStabilityReport = {+ probes: probed ? [...p.probes] : [], option_layout: p.layout,+ decision: soft === null ? null : decideReliability(soft, this.accept, this.escalate),+ soft_stability: soft, stability: probed ? rate(others) : null,+ rename_stability: probed ? rate(others.filter((v) => v.relabel)) : null,+ reorder_stability: probed ? rate(others.filter((v) => !v.relabel)) : null,+ n_variants: seen.length,+ distinct_choices: [...new Set(seen.flatMap((v) => v.choice === null ? [] : [v.choice]))].sort(compareCodePoints),+ variant_choices: Object.fromEntries(seen.map((v) => [v.name, v.choice])),+ variant_support: Object.fromEntries(seen.map((v) => [v.name, round4(v.support)])),+ };+ answer.reliability = report;+ }+ for (const key of internalKeys) delete answers[key];+ }+ const usage = result.usage;+ if (isRecord(usage)) {+ if (Array.isArray(usage.truncated_questions)) {+ usage.truncated_questions = usage.truncated_questions.filter((key) => !internalKeys.has(key));+ }+ if (isRecord(usage.options)) {+ usage.options = Object.fromEntries(Object.entries(usage.options).filter(([key]) => !internalKeys.has(key)));+ }+ }+ }+ }++ onError(ctx: PredictContext): void {+ this.plans.delete(ctx);+ }+}diff --git a/laya-ts/src/router.ts b/laya-ts/src/router.tsindex acd38eb7..68d4e2b3 100644--- a/laya-ts/src/router.ts+++ b/laya-ts/src/router.ts@@ -940,6 +940,7 @@ export class Router extends HookRegistry {if (group.maxLen !== null) agentOpts.maxLen = group.maxLen;if (group.headMaxLen !== null) agentOpts.headMaxLen = group.headMaxLen;if (group.lang !== null) agentOpts.lang = group.lang;+ markDefaultsRan(agentOpts);let batchResults: SystemOneResult[];if (typeof agent.predictBatch === "function") {batchResults = await agent.predictBatch(@@ -952,11 +953,13 @@ export class Router extends HookRegistry {// still work; their states simply share no forward pass.batchResults = [];for (const [, ctx] of group.items) {+ const singleOpts = group.lang === null ? {} : { lang: group.lang };+ markDefaultsRan(singleOpts);batchResults.push(await agent.systemOne(ctx.states[0],group.questions as Record<string, QuestionDef>,- group.lang === null ? undefined : { lang: group.lang },+ singleOpts,),);}diff --git a/laya-ts/tests/option-stability-integration.test.ts b/laya-ts/tests/option-stability-integration.test.tsnew file mode 100644index 00000000..bf0828b5--- /dev/null+++ b/laya-ts/tests/option-stability-integration.test.ts@@ -0,0 +1,199 @@+import { afterEach, describe, expect, it, vi } from "vitest";+import {+ Agent, Router, OptionStabilityHook, PredictContext, clearDefaultHooks, defaultTokenizer,+ setDefaultHooks, calibrateReliability, decideReliability, type Batch, type QuestionDef, type SessionProvider,+} from "../src/index.js";++afterEach(clearDefaultHooks);++// Exercise real validation, encoding, collation, decoding and hook dispatch. Only the+// model provider is synthetic: it selects the option span that still contains "approve".+function semanticAgent(extra: Record<string, unknown> = {}) {+ const batches: Batch[] = [];+ const tok = {+ ...defaultTokenizer(),+ encode(text: string) {+ return text.split(/[^\p{L}\p{N}_]+/u).filter(Boolean).map((word) => word === "approve" ? 900 : 901);+ },+ };+ const provider: SessionProvider = {+ async runEncoder(batch) {+ batches.push(batch);+ await Promise.resolve();+ return { lastHidden: batch.inputIds.map(() => [[0]]) };+ },+ async runHead(_hidden, batch) {+ const logits = batch.markerPos.map((positions, row) => positions.map((start, i) => {+ if (!batch.markerMask[row][i]) return -100;+ const next = positions[i + 1] || batch.inputIds[row].length;+ const sep = batch.inputIds[row].indexOf(tok.sepId, start);+ const end = sep < 0 ? next : Math.min(sep, next);+ return batch.inputIds[row].slice(start, end).includes(900) ? 6 : 0;+ }));+ return { logits, act: batch.qtype.map(() => [3, 0]) };+ },+ };+ return { agent: new Agent({ provider, tok, ...extra }), batches };+}++const hook = () => new OptionStabilityHook({ accept: 0.8, escalate: 0.6, variants: "fast", seed: "shared" });+const question = (criteria: unknown, order?: number[]): Record<string, QuestionDef> => ({+ action: { type: "choice", instructions: "Choose action", criteria, ...(order ? { option_order: order } : {}) },+});++describe("OptionStabilityHook through the local runtime", () => {+ it("requires explicit finite thresholds for JavaScript callers too", () => {+ for (const options of [{}, { accept: null }, { accept: undefined, escalate: null },+ { accept: NaN, escalate: 0 }, { accept: 0.5, escalate: 0.6 }, { accept: true, escalate: null }]) {+ expect(() => new OptionStabilityHook(options as never)).toThrow();+ }+ expect(() => decideReliability(0.9, undefined as never)).toThrow();+ expect(decideReliability(0.9, null, null)).toBeNull();+ expect(decideReliability(0.5, 0.8, null)).toBe("VERIFY");+ });++ it("rejects incomplete or invalid calibration evidence instead of silently dropping rows", () => {+ expect(() => calibrateReliability([0.8, 0.9], [true])).toThrow();+ expect(() => calibrateReliability([0.8], [true, false])).toThrow();+ expect(() => calibrateReliability([], [])).toThrow();+ expect(() => calibrateReliability([0.8], [1] as never)).toThrow();+ for (const score of [NaN, Infinity, -0.1, 1.1]) {+ expect(() => calibrateReliability([score], [true])).toThrow();+ }+ expect(() => calibrateReliability([0.8], [true], { targetAccuracy: NaN })).toThrow();+ expect(() => calibrateReliability([0.8], [true], { maxEscalateAccuracy: 1.1 })).toThrow();+ });++ it("cleans incomplete or malformed probe results without manufacturing reliability", () => {+ for (const bad of [undefined, null, { type: "choice", probabilities: { A: 1 } },+ { type: "choice", choice: "A", probabilities: { A: NaN } },+ { type: "choice", choice: "A", probabilities: { A: 1.1 } }]) {+ const stability = hook();+ const ctx = new PredictContext({ states: ["state"], questions: question(["approve", "reject"]) });+ stability.onPredictStart(ctx);+ const primary = { type: "choice", choice: "approve", probabilities: { approve: 1 } };+ const answers = { action: primary, "action::rel1": bad, "action::rel2": bad };+ ctx.results = [{ answers, usage: { options: { action: {}, "action::rel1": {} },+ truncated_questions: ["action", "action::rel2"] } }];+ expect(() => stability.onPredictEnd(ctx)).not.toThrow();+ expect(answers).toEqual({ action: primary });+ expect(primary).not.toHaveProperty("reliability");+ expect(ctx.results[0].usage).toEqual({ options: { action: {} }, truncated_questions: ["action"] });+ }+ });++ it.each([+ ["list", ["approve", "reject", "wait"]],+ ["empty descriptions", { approve: null, reject: "", wait: " " }],+ ["mixed", { approve: "approve request", reject: null, wait: { action: "wait" } }],+ ])("preserves %s semantics and caller inputs in a chunked batch", async (_name, criteria) => {+ const q = question(criteria, [2, 0, 1]);+ const original = structuredClone(q);+ const events: string[] = [];+ const { agent, batches } = semanticAgent({ hooks: [hook()] });+ const results = await agent.predictBatch(["first", "second"], q, {+ batchSize: 1,+ async onPredictStart() { await Promise.resolve(); events.push("start"); },+ async onPredictEnd(ctx) {+ await Promise.resolve();+ expect(Object.keys(ctx.results![0].answers as object)).toEqual(["action"]);+ events.push("end");+ },+ });+ expect(events).toEqual(["start", "end"]);+ expect(q).toEqual(original);+ expect(batches.map((b) => b.qtype.length)).toEqual([3, 3]);+ for (const result of results) {+ expect(Object.keys(result.answers)).toEqual(["action"]);+ expect(result.answers.action).toMatchObject({ choice: "approve", reliability: {+ decision: "ACCEPT", stability: 1, distinct_choices: ["approve"], n_variants: 3,+ variant_choices: { original: "approve", reversed: "approve", letters_reversed: "approve" },+ } });+ expect(result.usage.truncated_questions?.some((id) => id.includes("::rel"))).toBe(false);+ }+ });++ it("physically permutes integer-like labels and maps through caller option_order", async () => {+ const stability = new OptionStabilityHook({ accept: null, escalate: null, variants: ["reversed"], seed: "n" });+ const { agent, batches } = semanticAgent({ hooks: [stability] });+ const result = await agent.predict("state", question({ "1": "reject", "2": "approve", "10": "wait" }, [1, 2, 0]));+ expect(result.answers.action).toMatchObject({ choice: "2", reliability: { stability: 1, n_variants: 2 } });+ // Original physical order is approve, wait, reject; reversed+caller order is approve,+ // reject, wait. Non-semantic tokens coincide in the tokenizer, so inspect transformed+ // questions separately below as well as checking the actual decoder's label.+ expect(batches[0].markerPos).toHaveLength(2);+ const ctx = new PredictContext({ states: ["state"], questions: question({ "1": "reject", "2": "approve", "10": "wait" }) });+ stability.onPredictStart(ctx);+ const variant = ctx.questions["action::rel1"] as QuestionDef;+ expect(variant.option_order).toEqual([2, 1, 0]);+ stability.onError(ctx);+ });++ it("keeps concurrent calls isolated when a hook instance is shared", async () => {+ const stability = hook();+ const { agent } = semanticAgent({ hooks: [stability] });+ const [a, b] = await Promise.all([+ agent.predict("one", question(["approve", "reject"])),+ agent.predictBatch(["two", "three"], question(["wait", "approve", "reject"])),+ ]);+ for (const result of [a, ...b]) {+ expect(result.answers.action).toMatchObject({ choice: "approve", reliability: { stability: 1 } });+ expect(Object.keys(result.answers)).toEqual(["action"]);+ }+ });++ it("does not manufacture probe evidence for a cached primary-only answer", async () => {+ const stability = hook();+ const { agent, batches } = semanticAgent({ hooks: [stability] });+ const cached = { answers: { action: { type: "choice", choice: "approve", probabilities: { approve: 1 } } }, usage: {} };+ const result = await agent.predict("cached", question(["approve", "reject"]), {+ onPredictStart(ctx) { ctx.skip([structuredClone(cached)]); },+ });+ expect(result.answers.action).not.toHaveProperty("reliability");+ expect(batches).toHaveLength(0);+ expect((await agent.predict("fresh", question(["approve", "reject"]))).answers.action)+ .toHaveProperty("reliability.stability", 1);+ });++ it.each(["provider", "validation", "timeout"])("cleans the plan after a %s failure and preserves the error", async (mode) => {+ const stability = hook();+ let captured: PredictContext | undefined;+ const { agent } = semanticAgent({ hooks: [{ onPredictStart(ctx: PredictContext) { captured = ctx; } }, stability] });+ if (mode === "provider") vi.spyOn(agent.provider, "runEncoder").mockRejectedValueOnce(new Error("provider failed"));+ const q = question(["approve", "reject"]);+ if (mode === "validation") q.action.option_order = [0, 0];+ const opts = mode === "timeout" ? {+ hooksTimeout: 0.001, onPredictStart: () => new Promise<void>(() => {}),+ } : {};+ await expect(agent.predict("state", q, opts)).rejects.toThrow(+ mode === "provider" ? /provider failed/ : mode === "validation" ? /option_order/ : /exceeded/,+ );+ expect(captured!.error).not.toBeNull();+ // Calling end again on the captured failed context must not find an old run plan.+ const answers = Object.fromEntries(Object.entries(captured!.questions).map(([qid, definition]) => {+ const criteria = (definition as QuestionDef).criteria;+ const key = Array.isArray(criteria) ? criteria[0] : Object.keys(criteria as object)[0];+ return [qid, { type: "choice", choice: key, probabilities: { [key]: 1 } }];+ }));+ captured!.results = [{ answers, usage: {} }];+ stability.onPredictEnd(captured!);+ expect(answers.action).not.toHaveProperty("reliability");+ expect((await agent.predict("recovered", question(["approve", "reject"]))).answers.action)+ .toHaveProperty("reliability.stability", 1);+ });++ it.each([false, true])("runs a default stability hook once through Router batch, fallback=%s", async (fallback) => {+ const { agent } = semanticAgent();+ const loaded = fallback ? { systemOne: agent.systemOne.bind(agent), cfg: agent.cfg } : agent;+ const router = new Router({ default: "english", loader: (() => loaded) as never });+ setDefaultHooks(hook());+ const results = await router.predictBatch([+ { state: "one", questions: question(["approve", "reject"]) },+ { state: "two", questions: question(["approve", "reject"]) },+ ]);+ for (const result of results) {+ expect(result.answers.action).toMatchObject({ choice: "approve", reliability: { stability: 1 } });+ expect(Object.keys(result.answers)).toEqual(["action"]);+ }+ });+});diff --git a/laya-ts/tests/option-stability-parity.test.ts b/laya-ts/tests/option-stability-parity.test.tsnew file mode 100644index 00000000..5389c859--- /dev/null+++ b/laya-ts/tests/option-stability-parity.test.ts@@ -0,0 +1,158 @@+import { readFileSync } from "node:fs";+import { describe, expect, it } from "vitest";+import {+ OptionStabilityHook,+ PredictContext,+ calibrateReliability,+ decideReliability,+ makeVariants,+ type CalibrationOptions,+ type OptionStabilityOptions,+ type OptionStabilityReport,+ type StabilityVariant,+ type VariantOptions,+} from "../src/index.js";+import { renderOptions } from "../src/common.js";++type Question = {+ type: "choice";+ instructions: string;+ criteria: string[] | Record<string, unknown>;+ option_order?: number[];+};+type Result = {+ answers: Record<string, {+ type: "choice";+ choice: string;+ probabilities: Record<string, number>;+ reliability?: OptionStabilityReport;+ }>;+ usage: Record<string, unknown>;+};+type HookCase = {+ name: string;+ states: unknown[];+ questions: Record<string, Question>;+ layout: "parallel" | "sequential";+ options: OptionStabilityOptions;+ expanded: { key: string; rendered: string[] }[];+ model_results: Result[];+ expected_results: Result[];+};+type Fixture = {+ schema: string;+ variant_cases: {+ name: string; labels: string[]; seed_key: string; options: VariantOptions; expected: StabilityVariant[];+ }[];+ hook_cases: HookCase[];+ calibration_cases: {+ name: string; scores: number[]; correct: boolean[]; options: CalibrationOptions;+ thresholds: [number, number]; decisions: (string | null)[];+ }[];+ decision_cases: { score: number; accept: number | null; escalate: number | null; expected: string | null }[];+ pipeline: {+ cases: string[]; correct: boolean[]; options: CalibrationOptions;+ thresholds: [number, number]; decisions: (string | null)[];+ };+};++// Generated by the actual Python hook and helpers, without model weights. The+// fixture records the exact reference commits and file hashes, including pending+// #1286/#1289 semantics. No Python installation is needed to run this TS suite.+const fixture = JSON.parse(readFileSync(new URL("../fixtures/option-stability.json", import.meta.url), "utf8")) as Fixture;++function physicalOptions(question: Question): string[] {+ const criteria = Array.isArray(question.criteria)+ ? Object.fromEntries(question.criteria.map((label) => [label, null]))+ : question.criteria;+ const rendered = renderOptions({ t: "choice", ins: question.instructions, crit: criteria });+ return (question.option_order ?? rendered.map((_, index) => index)).map((index) => rendered[index]);+}++function runHook(c: HookCase): Result[] {+ const questions = structuredClone(c.questions);+ const hook = new OptionStabilityHook(c.options);+ const context = new PredictContext({ states: structuredClone(c.states), questions,+ agent: { cfg: { option_layout: c.layout } } });+ hook.onPredictStart(context);+ expect(questions).toEqual(c.questions);+ // Compare what reaches each physical option slot. Integer-like JS object keys+ // reorder themselves, so raw criteria insertion order is not a valid oracle;+ // the TS hook uses option_order to realize the Python probe permutation.+ expect(Object.entries(context.questions).map(([key, q]) => ({ key, rendered: physicalOptions(q as Question) })))+ .toEqual(c.expanded);+ context.results = structuredClone(c.model_results);+ hook.onPredictEnd(context);+ expect(context.results).toEqual(c.expected_results);+ expect(questions).toEqual(c.questions);+ // A completed plan must be consumed, so a repeated end event cannot apply the+ // same mapping to already-cleaned output or produce an unprobed report.+ hook.onPredictEnd(context);+ expect(context.results).toEqual(c.expected_results);+ return context.results as Result[];+}++describe("Python option stability fixtures", () => {+ it("uses the supported fixture schema", () => {+ expect(fixture.schema).toBe("laya-option-stability/1");+ });++ describe("CRC32 / Python integer-seeded shuffle / filtering / deduplication", () => {+ for (const c of fixture.variant_cases) {+ it(c.name, () => {+ const labels = [...c.labels];+ expect(makeVariants(labels, c.seed_key, c.options)).toEqual(c.expected);+ expect(labels).toEqual(c.labels);+ expect(makeVariants(labels, c.seed_key, c.options)).toEqual(c.expected);+ });+ }+ });++ describe("expanded semantics, label mapping, reporting and cleanup", () => {+ for (const c of fixture.hook_cases) {+ it(c.name, () => {+ const results = runHook(c);+ for (const result of results) {+ const report = result.answers.q.reliability!;+ if (report.soft_stability === null) {+ expect(report.decision).toBeNull();+ expect(report.probes).toEqual([]);+ } else {+ // Use the persisted public score, not rounded variant_support or an+ // independently reconstructed mean, to replay the actual decision.+ const saved = JSON.parse(JSON.stringify(report)) as OptionStabilityReport;+ expect(decideReliability(saved.soft_stability!, c.options.accept, c.options.escalate))+ .toBe(report.decision);+ }+ }+ });+ }+ });++ describe("raw calibration cohorts and boundary replay", () => {+ for (const c of fixture.calibration_cases) {+ it(c.name, () => {+ const thresholds = calibrateReliability(c.scores, c.correct, c.options);+ expect(thresholds).toEqual(c.thresholds);+ expect(c.scores.map((score) => decideReliability(score, ...thresholds))).toEqual(c.decisions);+ });+ }+ it("matches all categorical boundaries, including adjacent representable numbers", () => {+ for (const c of fixture.decision_cases) {+ expect(decideReliability(c.score, c.accept, c.escalate)).toBe(c.expected);+ }+ });+ it("fits the persisted runtime reports and retains cuts that share four decimal digits", () => {+ const pipeline = fixture.pipeline;+ const scores = pipeline.cases.map((name) => {+ const c = fixture.hook_cases.find((item) => item.name === name)!;+ return runHook(c)[0].answers.q.reliability!.soft_stability!;+ });+ expect(new Set(scores).size).toBe(3);+ const savedScores = JSON.parse(JSON.stringify(scores)) as number[];+ const thresholds = calibrateReliability(savedScores, pipeline.correct, pipeline.options);+ expect(thresholds).toEqual(pipeline.thresholds);+ expect(savedScores.map((score) => decideReliability(score, ...thresholds))).toEqual(pipeline.decisions);+ });+ });+});diff --git a/laya-ts/tests/package-e2e.mjs b/laya-ts/tests/package-e2e.mjsindex ab60201a..3310ce38 100644--- a/laya-ts/tests/package-e2e.mjs+++ b/laya-ts/tests/package-e2e.mjs@@ -107,7 +107,8 @@ test("the packed module installs and works in an external ESM TypeScript projectawait writeFile(join(consumerDir, "runtime.mjs"),`import assert from "node:assert/strict";-import { Agent, VERSION, predictTournament, DEFAULT_TOURNAMENT_GROUP } from "laya-ts";+import { Agent, VERSION, predictTournament, DEFAULT_TOURNAMENT_GROUP,+ OptionStabilityHook, calibrateReliability, decideReliability } from "laya-ts";let encoderCalls = 0;let headCalls = 0;@@ -116,9 +117,9 @@ const provider = {encoderCalls += 1;return { lastHidden: [[[0]]] };},- async runHead() {+ async runHead(_hidden, batch) {headCalls += 1;- return { logits: [[4, 0]], act: [[3, 0]] };+ return { logits: batch.qtype.map(() => [4, 0]), act: batch.qtype.map(() => [3, 0]) };},};const agent = new Agent({ provider });@@ -142,6 +143,14 @@ const tournament = await predictTournament(agent, "charged twice", {});assert.equal(tournament.answers.intent.choice, "refund");assert.deepEqual(tournament.tournament.intent, { labels: ["refund", "other"], n: 2, rounds: 0 });+const stable = await agent.predict("charged twice", {+ intent: { type: "choice", instructions: "Pick a route", criteria: ["refund", "other"] },+}, { hooks: [new OptionStabilityHook({ accept: null, escalate: null, variants: "fast" })] });+assert.deepEqual(Object.keys(stable.answers), ["intent"]);+assert.equal(stable.answers.intent.reliability.n_variants, 3);+assert.equal(stable.answers.intent.reliability.decision, null);+const cuts = calibrateReliability([0.80001, 0.80002], [false, true], { targetAccuracy: 1 });+assert.equal(decideReliability(0.80002, ...cuts), "ACCEPT");`,);await run(process.execPath, ["runtime.mjs"], consumerDir);@@ -153,6 +162,8 @@ assert.deepEqual(tournament.tournament.intent, { labels: ["refund", "other"], n:VERSION,predictTournament,DEFAULT_TOURNAMENT_GROUP,+ OptionStabilityHook,+ type OptionStabilityReport,type QuestionDef,type SessionProvider,type SystemOneResult,@@ -187,6 +198,13 @@ void prediction;const tournament: Promise<SystemOneResult & { tournament: Record<string, TournamentMeta> }> =predictTournament(new Agent({ provider }), "charged twice", questions, DEFAULT_TOURNAMENT_GROUP);void tournament;+const stability = new OptionStabilityHook({ accept: null, escalate: null });+new Agent({ provider }).addHook(stability);+async function reliability(): Promise<OptionStabilityReport | undefined> {+ const result = await prediction;+ return result.answers.intent.type === "choice" ? result.answers.intent.reliability : undefined;+}+void reliability;`,);await writeFile(diff --git a/laya-ts/tests/router-batch.test.ts b/laya-ts/tests/router-batch.test.tsindex 74ecb7a0..a6e5ad78 100644--- a/laya-ts/tests/router-batch.test.ts+++ b/laya-ts/tests/router-batch.test.ts@@ -1,10 +1,40 @@import { describe, expect, it, vi } from "vitest";-import { Agent, Router } from "../src/index.js";+import { Agent, Router, clearDefaultHooks, setDefaultHooks } from "../src/index.js";import type { BatchRequest } from "../src/index.js";// TS mirror of tests/test_router_batch.py and the orchestration part of tests/test_batch.py.const Q = { intent: { type: "noul", instructions: "Relevant?" } } as const;++// A stateful default hook must not run again inside the Agent after the Router has+// already expanded its questions (as OptionStabilityHook does).+it.each([false, true])("runs process defaults once per request with fallback=%s", async (fallback) => {+ const starts: unknown[] = [];+ const ends: unknown[] = [];+ const agent = new Agent({ provider: {+ async runEncoder(batch) { return { lastHidden: batch.inputIds.map(() => [[0]]) }; },+ async runHead(_hidden, batch) {+ return { logits: batch.qtype.map(() => [2, 0]), act: batch.qtype.map(() => [3, 0]) };+ },+ } });+ const loaded = fallback ? { systemOne: agent.systemOne.bind(agent) } : agent;+ const router = new Router({ default: "english", loader: (() => loaded) as never });+ setDefaultHooks({+ onPredictStart(ctx) { starts.push([...ctx.states]); },+ onPredictEnd(ctx) { ends.push([...ctx.states]); },+ });+ try {+ const results = await router.predictBatch([+ { state: "first request", questions: Q },+ { state: "second request", questions: Q },+ ]);+ expect(results).toHaveLength(2);+ expect(starts).toEqual([["first request"], ["second request"]]);+ expect(ends).toEqual(starts);+ } finally {+ clearDefaultHooks();+ }+});const req = (state: unknown, overrides: Record<string, unknown> = {}): BatchRequest =>({ state, questions: Q, ...overrides }) as unknown as BatchRequest;diff --git a/laya-ts/tests/types/option-stability.ts b/laya-ts/tests/types/option-stability.tsnew file mode 100644index 00000000..24af48dd--- /dev/null+++ b/laya-ts/tests/types/option-stability.ts@@ -0,0 +1,31 @@+import {+ Agent, OptionStabilityHook, calibrateReliability, decideReliability, makeVariants,+ type OptionStabilityOptions, type OptionStabilityReport, type ReliabilityDecision,+} from "../../src/index.js";++declare const agent: Agent;+const opts: OptionStabilityOptions = { accept: null, escalate: null, seed: "shared" };+agent.addHook(new OptionStabilityHook(opts));+const cuts: [number, number] = calibrateReliability([0.8, 0.9], [false, true], { targetAccuracy: 1 });+const decision: ReliabilityDecision | null = decideReliability(0.9, ...cuts);+const variants = makeVariants(["approve", "reject"], "seed", { nShuffles: 2 });+const labels: string[] = variants[0].order;++async function inspect() {+ const result = await agent.predict("state", { q: { type: "choice", criteria: ["approve", "reject"] } });+ if (result.answers.q.type === "choice") {+ const report: OptionStabilityReport | undefined = result.answers.q.reliability;+ const score: number | null | undefined = report?.soft_stability;+ return score;+ }+}++// @ts-expect-error explicit thresholds are required+new OptionStabilityHook({ seed: "missing thresholds" });+// @ts-expect-error thresholds must be numeric or null+new OptionStabilityHook({ accept: "0.9", escalate: null });+// @ts-expect-error invalid probe name+new OptionStabilityHook({ accept: null, escalate: null, probes: "confidence" });+void decision;+void labels;+void inspect;diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.pyindex 1a5840d9..772aff15 100644--- a/tests/test_hooks_api.py+++ b/tests/test_hooks_api.py@@ -1594,6 +1594,14 @@ from laya import hooks as _hooks_mod # noqa: E402check_true("hooks module exports compose_hooks", callable(_hooks_mod.compose_hooks))+# The local TypeScript stability port consumes these reference helper contracts.+check("stability/make_variants reference parameters", list(sig(_hooks_mod.make_variants)),+ ["labels", "seed_key", "n_shuffles", "relabel", "include", "reorder"])+check("stability/calibrate_reliability reference parameters", list(sig(_hooks_mod.calibrate_reliability)),+ ["soft_scores", "correct", "target_accuracy", "max_escalate_accuracy"])+check("stability/decide_reliability reference parameters", list(sig(_hooks_mod.decide_reliability)),+ ["soft_stability", "accept", "escalate"])+_patterns_md = os.path.join(REPO, "docs", "hooks", "patterns.md")with open(_patterns_md, encoding="utf-8", newline="") as _pmf:_patterns_text = _pmf.read().replace("\r\n", "\n")