Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
feat(ts): add tournament selection for large choice sets
xp987ac82e785a
diff --git a/laya-ts/README.md b/laya-ts/README.mdindex 713f7e56..864c5e60 100644--- a/laya-ts/README.md+++ b/laya-ts/README.md@@ -178,6 +178,55 @@ const out = await predictShortlist(agent, state, questions, embedFn, 20);// embedFnFromAgent(agent) mean-pools the loaded encoder; a dedicated bi-encoder usually shortlists better.```+### Tournament (without an embedder)++For a large tool or intent catalog, `predictTournament` lets the decision model narrow the+options itself. It ports Python's `predict_tournament` to the local Node/browser runtime:++```ts+import { predictTournament } from "laya-ts";++const out = await predictTournament(agent, state, questions);+console.log(out.answers.intent);+console.log(out.tournament.intent); // { labels: [...finalists], n: originalCount, rounds: 1 }++// Pick a group size and forward options to every prediction, including the final one.+const routed = await predictTournament(router, state, questions, 16, { model: "english" });+```++Each oversized choice is split into balanced groups of at most 16 labels. All groups from+all questions share one `predict` call per round; their winners advance until each choice+fits. Up to 256 labels therefore take one elimination call and one final call. Small choices+and non-choice questions reach the final call unchanged, and a request needing no+elimination makes exactly one call. The helper does not mutate the supplied questions.++The returned `tournament` metadata covers every choice, including `rounds: 0` for a+passthrough. Probabilities and confidences describe **only the finalists**; `usage`,+`routing`, and other prediction fields come from the final call, not all rounds combined.+Hooks run for every underlying call. Calibrate thresholds on the strategy you actually use;+a threshold fitted on the full catalog does not automatically transfer to a tournament.+Widening the head budget or using a dedicated shortlist embedder can still be a better fit.++A narrowed choice drops its original `option_order` with one warning per question, matching+Python. Passthrough definitions keep it. This does not add `option_order` handling to the+TypeScript `Agent`, whose prediction path does not currently apply that field.++For a reproducible local comparison, build the package and export a checkpoint as above,+then run from the repository root:++```bash+python laya-ts/scripts/tournament_parity.py --model-dir CHECKPOINT --onnx-dir ./model \+ --fixture ./tournament-python.json+node laya-ts/examples/tournament.mjs --model-dir ./model \+ --fixture ./tournament-python.json --report ./tournament-node.json+```++The Python script records fresh predictions for fixed inputs; Node independently runs+the same inputs through the split ONNX export. The check compares each round's questions,+winners, probabilities and final metadata, including a two-round case. Reports include+artifact hashes, call counts and warmed timings. These few cases check runtime parity;+they do not establish accuracy gains or production latency. Model files stay local.+## Per-language calibration (`lang_temperatures`)Port of the Python `Agent(lang_temperatures=...)` knob. A language override replaces thediff --git a/laya-ts/examples/tournament.mjs b/laya-ts/examples/tournament.mjsnew file mode 100644index 00000000..ca2c48a6--- /dev/null+++ b/laya-ts/examples/tournament.mjs@@ -0,0 +1,161 @@+// Run after scripts/tournament_parity.py; all model files must already be local.+// node laya-ts/examples/tournament.mjs --model-dir SPLIT_EXPORT \+// --fixture /tmp/tournament-python.json --report /tmp/tournament-node.json+// Java contributes the input scenarios only. This runs the TS split-ONNX model.+import assert from "node:assert/strict";+import { createHash } from "node:crypto";+import { readFile, stat, mkdir, writeFile } from "node:fs/promises";+import { dirname, isAbsolute, relative, resolve, sep } from "node:path";+import { parseArgs } from "node:util";+import { Agent, predictTournament } from "../dist/index.js";++const { values } = parseArgs({ options: {+ "model-dir": { type: "string" }, fixture: { type: "string" },+ report: { type: "string" }, tolerance: { type: "string", default: "0.001" },+} });+assert(values["model-dir"] && values.fixture && values.report,+ "usage: tournament.mjs --model-dir LOCAL_SPLIT_EXPORT --fixture PYTHON.json --report NODE.json");+const modelDir = resolve(values["model-dir"]);+assert((await stat(modelDir)).isDirectory(), "--model-dir must be an existing local directory");+const reportPath = resolve(values.report);+assert.notEqual(resolve(values.fixture), reportPath, "report must not overwrite fixture");+const reportRelative = relative(modelDir, reportPath);+assert(isAbsolute(reportRelative) || reportRelative.startsWith(`..${sep}`),+ "--report must be outside the model directory");+const tolerance = Number(values.tolerance);+assert(Number.isFinite(tolerance) && tolerance > 0, "tolerance must be a positive finite number");+const fixtureBytes = await readFile(values.fixture);+const fixture = JSON.parse(fixtureBytes.toString("utf8"));+assert.equal(fixture.schema, "laya-ts-tournament-parity/1");+assert(Number.isInteger(fixture.repeats) && fixture.repeats > 0, "invalid repeat count");+assert(Array.isArray(fixture.cases) && fixture.cases.length > 0, "fixture has no cases");+assert.equal(new Set(fixture.cases.map(item => item.name)).size, fixture.cases.length,+ "fixture case names must be unique");+for (const item of fixture.cases) {+ assert(typeof item.name === "string" && item.name.length > 0, "invalid case name");+ assert.equal(item.plain?.calls?.length, 1, "plain reference must contain one real call");+ assert(Array.isArray(item.tournament?.calls) && item.tournament.calls.length > 0,+ "tournament reference has no calls");+}+for (const name of ["encoder.onnx", "head.onnx", "rl_agent_config.json", "tokenizer.json"]) {+ assert.match(fixture.onnx_sha256?.[name] ?? "", /^[0-9a-f]{64}$/, `missing digest: ${name}`);+}+const pythonOptions = fixture.predict_kwargs;+assert.deepEqual(Object.keys(pythonOptions).sort(), ["head_max_len", "lang", "max_len", "min_confidence"]);+const options = { lang: pythonOptions.lang, maxLen: pythonOptions.max_len,+ headMaxLen: pythonOptions.head_max_len, minConfidence: pythonOptions.min_confidence };++function recording(agent) {+ return { calls: [], async predict(state, questions, kwargs = {}) {+ const call = { state: structuredClone(state), questions: structuredClone(questions),+ kwargs: { lang: kwargs.lang, max_len: kwargs.maxLen, head_max_len: kwargs.headMaxLen,+ min_confidence: kwargs.minConfidence } };+ const started = performance.now();+ const result = await agent.predict(state, questions, kwargs);+ this.calls.push({ ...call, answers: structuredClone(result.answers),+ usage: structuredClone(result.usage), elapsed_ms: performance.now() - started });+ return result;+ } };+}++async function measure(agent, run) {+ await run(recording(agent)); // warm this path, outside the measured samples+ const elapsed = [];+ let calls, result;+ for (let i = 0; i < fixture.repeats; i++) {+ const recorder = recording(agent);+ const started = performance.now();+ result = await run(recorder);+ elapsed.push(performance.now() - started);+ if (calls === undefined) calls = recorder.calls;+ else assert.deepEqual(recorder.calls.map(c => c.answers), calls.map(c => c.answers),+ "TS outputs changed across repeated calls");+ }+ const sorted = [...elapsed].sort((a, b) => a - b);+ const mid = Math.floor(sorted.length / 2);+ const median = sorted.length % 2 ? sorted[mid] : (sorted[mid - 1] + sorted[mid]) / 2;+ return { calls, result, model_calls: calls.length, elapsed_ms: elapsed, median_ms: median };+}++const errors = [];+let maxDifference = 0;+function compare(expected, actual, path) {+ if (typeof expected === "number") {+ const difference = Math.abs(expected - actual);+ if (typeof actual !== "number" || !Number.isFinite(actual) || difference > tolerance) {+ errors.push(`${path}: expected ${expected}, got ${actual}`);+ }+ if (Number.isFinite(difference)) maxDifference = Math.max(maxDifference, difference);+ } else if (expected && typeof expected === "object") {+ if (!actual || typeof actual !== "object" ||+ JSON.stringify(Object.keys(expected)) !== JSON.stringify(Object.keys(actual))) {+ errors.push(`${path}: object keys or their order differ`);+ return;+ }+ for (const key of Object.keys(expected)) compare(expected[key], actual[key], `${path}.${key}`);+ } else if (expected !== actual) errors.push(`${path}: expected ${expected}, got ${actual}`);+}++function compareAnswers(expected, actual, path) {+ if (JSON.stringify(Object.keys(expected)) !== JSON.stringify(Object.keys(actual ?? {}))) {+ errors.push(`${path}: answer IDs or their order differ`);+ return;+ }+ for (const [qid, answer] of Object.entries(expected)) {+ // Runtime-specific model names and abstention annotations are outside this port.+ for (const field of ["type", "choice", "noul", "score", "probabilities",+ "confidence", "answer_confidence", "action"]) {+ if (field in answer) compare(answer[field], actual[qid]?.[field], `${path}.${qid}.${field}`);+ }+ compare(answer.low_confidence === true, actual[qid]?.low_confidence === true,+ `${path}.${qid}.low_confidence`);+ }+}++function compareRun(expected, actual, path) {+ if (expected.calls.length !== actual.calls.length) errors.push(`${path}: model call count differs`);+ for (let i = 0; i < Math.min(expected.calls.length, actual.calls.length); i++) {+ const want = expected.calls[i], got = actual.calls[i];+ // Order is significant for state serialization, options, and generated group IDs.+ for (const field of ["state", "questions", "kwargs"]) {+ if (JSON.stringify(want[field]) !== JSON.stringify(got[field])) {+ errors.push(`${path}.calls[${i}].${field}: exact input mismatch`);+ }+ }+ compareAnswers(want.answers, got.answers, `${path}.calls[${i}].answers`);+ }+ compareAnswers(expected.result.answers, actual.result.answers, `${path}.final.answers`);+ if (expected.result.tournament) {+ // Counts and finalist order are exact, not floating-point comparisons.+ if (JSON.stringify(expected.result.tournament) !== JSON.stringify(actual.result.tournament)) {+ errors.push(`${path}.final.tournament: metadata differs`);+ }+ }+}++const agent = await Agent.load(modelDir, { device: "cpu", numThreads: 2,+ expectedSha256: fixture.onnx_sha256 });+const results = [];+for (const item of fixture.cases) {+ const { state, questions } = item;+ const plain = await measure(agent, recorder => recorder.predict(state, questions, options));+ const tournament = await measure(agent, recorder =>+ predictTournament(recorder, state, questions, item.group_size ?? 16, options));+ compareRun(item.plain, plain, `${item.name}.plain`);+ compareRun(item.tournament, tournament, `${item.name}.tournament`);+ results.push({ name: item.name, plain, tournament });+ console.log(JSON.stringify({ case: item.name, plain_ms: plain.median_ms,+ tournament_ms: tournament.median_ms, model_calls: tournament.model_calls }));+}+const report = { schema: "laya-ts-tournament-parity-result/1",+ fixture_sha256: createHash("sha256").update(fixtureBytes).digest("hex"),+ onnx_sha256: fixture.onnx_sha256, environment: { node: process.version,+ platform: process.platform, arch: process.arch, device: "cpu", num_threads_requested: 2 },+ tolerance, max_absolute_difference: maxDifference, passed: errors.length === 0,+ errors, cases: results,+ timing_note: "One warmup per case and strategy; elapsed samples exclude loading and hashing. Small parity experiment, not an accuracy or throughput benchmark." };+await mkdir(dirname(reportPath), { recursive: true });+await writeFile(reportPath, JSON.stringify(report, null, 2) + "\n", "utf8");+console.log(JSON.stringify({ passed: report.passed, max_absolute_difference: maxDifference,+ errors: errors.length, report: reportPath }));+if (errors.length) process.exitCode = 1;diff --git a/laya-ts/scripts/tournament_parity.py b/laya-ts/scripts/tournament_parity.pynew file mode 100644index 00000000..c1fe8102--- /dev/null+++ b/laya-ts/scripts/tournament_parity.py@@ -0,0 +1,155 @@+"""Record a local PyTorch reference for the TypeScript tournament example.++Export the same checkpoint with laya-ts/scripts/export_onnx.py first, then run from the+repository root (with the checkout installed):++ python laya-ts/scripts/tournament_parity.py --model-dir CHECKPOINT \+ --onnx-dir SPLIT_EXPORT --fixture /tmp/tournament-python.json+ node laya-ts/examples/tournament.mjs --model-dir SPLIT_EXPORT \+ --fixture /tmp/tournament-python.json --report /tmp/tournament-node.json++Only inputs are reused from the Java golden fixture; its predictions and its+single-graph ONNX results are never used as the reference. Both runtimes execute+real models independently. This is a small parity experiment, not an accuracy or+throughput benchmark. Each path is warmed once before its measured call(s).+"""+import argparse+import hashlib+import json+import os+import platform+import statistics+import time+from copy import deepcopy+from pathlib import Path++SCHEMA = "laya-ts-tournament-parity/1"+CASES = Path(__file__).resolve().parents[2] / "laya-java/fixtures/tournament_golden.json"+OPTIONS = {"lang": "en", "max_len": 512, "head_max_len": 192, "min_confidence": 0.9}+++def digest(path):+ value = hashlib.sha256()+ with path.open("rb") as handle:+ for chunk in iter(lambda: handle.read(1024 * 1024), b""):+ value.update(chunk)+ return value.hexdigest()+++class Recording:+ def __init__(self, agent):+ self.agent, self.calls = agent, []++ def predict(self, state, questions, **kwargs):+ call = {"state": deepcopy(state), "questions": deepcopy(questions),+ "kwargs": deepcopy(kwargs)}+ started = time.perf_counter()+ result = self.agent.predict(state, questions, **kwargs)+ call.update(answers=deepcopy(result["answers"]), usage=deepcopy(result["usage"]),+ elapsed_ms=(time.perf_counter() - started) * 1000)+ self.calls.append(call)+ return result+++def measure(agent, run, repeats):+ run(Recording(agent)) # warm this path; excluded from reported timings+ elapsed, reference, result = [], None, None+ for _ in range(repeats):+ recorder = Recording(agent)+ started = time.perf_counter()+ result = run(recorder)+ elapsed.append((time.perf_counter() - started) * 1000)+ if reference is None:+ reference = recorder.calls+ elif [call["answers"] for call in recorder.calls] != [call["answers"] for call in reference]:+ raise RuntimeError("the reference changed across repeated calls")+ return {"calls": reference, "result": result, "model_calls": len(reference),+ "elapsed_ms": elapsed, "median_ms": statistics.median(elapsed)}+++def main():+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--model-dir", type=Path, required=True, help="local safetensors checkpoint")+ parser.add_argument("--onnx-dir", type=Path, required=True, help="split export of that checkpoint")+ parser.add_argument("--fixture", type=Path, required=True, help="write the new Python reference JSON")+ parser.add_argument("--repeats", type=int, default=1, help="measured calls per path after one warmup")+ args = parser.parse_args()+ checkpoint, exported = args.model_dir.resolve(), args.onnx_dir.resolve()+ if not checkpoint.is_dir() or not exported.is_dir():+ parser.error("--model-dir and --onnx-dir must be existing local directories")+ if args.repeats < 1:+ parser.error("--repeats must be positive")+ if args.fixture.resolve() == CASES or args.fixture.resolve().is_relative_to(exported):+ parser.error("--fixture must be outside the export directory and must not overwrite the input cases")+ required = ("encoder.onnx", "head.onnx", "rl_agent_config.json", "tokenizer.json")+ for name in required:+ if not (exported / name).is_file():+ parser.error("split export is missing " + name)+ # Agent reads tokenizer/, while the split exporter may use a root-level copy.+ source_names = ("model.safetensors", "rl_agent_config.json", "tokenizer/tokenizer.json")+ for name in source_names:+ if not (checkpoint / name).is_file():+ parser.error("local Python checkpoint is missing " + name)+ source_hashes = {name: digest(checkpoint / name) for name in source_names}+ for name in ("encoder/config.json", "tokenizer.json"):+ source_hashes[name] = digest(checkpoint / name) if (checkpoint / name).is_file() else None+ # Include external tensors as well as the graphs, regardless of their filenames.+ export_hashes = {path.relative_to(exported).as_posix(): digest(path)+ for path in sorted(exported.rglob("*")) if path.is_file()}+ for source, target in (("rl_agent_config.json", "rl_agent_config.json"),+ ("tokenizer/tokenizer.json", "tokenizer.json")):+ if source_hashes[source] != export_hashes[target]:+ parser.error("checkpoint and split export differ: " + target)++ source_cases = json.loads(CASES.read_text(encoding="utf-8"))["cases"]+ cases = [{key: deepcopy(case[key]) for key in ("name", "state", "questions", "group_size")}+ for case in source_cases]+ # The same 40 labels at size 4 exercise two narrowing rounds without new test text.+ two_rounds = deepcopy(next(case for case in cases if case["name"] == "banking-default-group"))+ two_rounds.update(name="banking-two-rounds", group_size=4)+ cases.append(two_rounds)++ # Loading must use local checkpoint/cache files even when optional config is absent.+ os.environ["HF_HUB_OFFLINE"] = "1"+ os.environ["TRANSFORMERS_OFFLINE"] = "1"++ import torch+ import transformers++ import laya++ torch.set_num_threads(2)+ torch.manual_seed(0)+ document = {+ "schema": SCHEMA,+ "checkpoint_sha256": source_hashes,+ "onnx_sha256": export_hashes,+ "case_source_sha256": digest(CASES),+ "environment": {"python": platform.python_version(), "platform": platform.platform(),+ "laya": laya.__version__, "torch": str(torch.__version__),+ "transformers": transformers.__version__, "torch_threads": 2,+ "device": "cpu", "backend": "eager"},+ "predict_kwargs": OPTIONS,+ "repeats": args.repeats,+ "cases": [],+ }+ with laya.load(str(checkpoint), device="cpu", backend="eager") as agent:+ for case in cases:+ state, questions = case["state"], case["questions"]+ group = 16 if case["group_size"] is None else case["group_size"]+ plain = measure(agent, lambda recorder, s=state, q=questions: recorder.predict(s, q, **OPTIONS),+ args.repeats)+ tournament = measure(+ agent, lambda recorder, s=state, q=questions, g=group: laya.predict_tournament(+ recorder, s, q, group_size=g, **OPTIONS), args.repeats)+ document["cases"].append(dict(case, plain=plain, tournament=tournament))+ print(json.dumps({"case": case["name"], "plain_ms": plain["median_ms"],+ "tournament_ms": tournament["median_ms"],+ "model_calls": tournament["model_calls"]}), flush=True)+ args.fixture.parent.mkdir(parents=True, exist_ok=True)+ args.fixture.write_text(json.dumps(document, ensure_ascii=False, indent=2, allow_nan=False) + "\n",+ encoding="utf-8")+++if __name__ == "__main__":+ main()diff --git a/laya-ts/src/index.ts b/laya-ts/src/index.tsindex a9dd789d..fec9d250 100644--- a/laya-ts/src/index.ts+++ b/laya-ts/src/index.ts@@ -17,8 +17,8 @@ export { createNodeProvider, createWebProvider, feed, feedHead, loadNodeBundle,export type { Batch, SessionProvider, ProviderOptions, NodeBundle, WebBundle } from "./providers.js";export { Router, normaliseName, DEFAULT_MODELS, digestsFromEnv, _digestsFromEnv } from "./router.js";export type { RoutedResult, RouteDecision, ModelName, ModelSpec, BatchRequest, PredictBatchOptions, RouterOptions, RouteOptions } from "./router.js";-export { shortlistChoice, predictShortlist, embedFnFromAgent, DEFAULT_SHORTLIST_K } from "./shortlist.js";-export type { EmbedFn, ShortlistMeta } from "./shortlist.js";+export { shortlistChoice, predictShortlist, predictTournament, embedFnFromAgent, DEFAULT_SHORTLIST_K, DEFAULT_TOURNAMENT_GROUP } from "./shortlist.js";+export type { EmbedFn, ShortlistMeta, TournamentMeta } from "./shortlist.js";export { analyse, isEnglish, guessLatinLanguage, detectScript } from "./lang.js";export type { AnalyseResult, LatinProfile } from "./lang.js";export { cleanEmailBody, emailState } from "./email.js";diff --git a/laya-ts/src/shortlist.ts b/laya-ts/src/shortlist.tsindex f856ae96..9571d922 100644--- a/laya-ts/src/shortlist.ts+++ b/laya-ts/src/shortlist.ts@@ -3,6 +3,7 @@ import type { Batch } from "./providers.js";import type { QuestionDef, SystemOneResult } from "./agent.js";export const DEFAULT_SHORTLIST_K = 20;+export const DEFAULT_TOURNAMENT_GROUP = 16;export type EmbedFn = (texts: string[]) => Promise<number[][]> | number[][];@@ -14,6 +15,12 @@ export interface ShortlistMeta {passthrough: boolean;}+export interface TournamentMeta {+ labels: string[];+ n: number;+ rounds: number;+}+function checkK(k: unknown): number {if (typeof k !== "number" || !Number.isInteger(k) || k < 1) {throw new Error(`k must be a positive integer, got ${JSON.stringify(k)}`);@@ -177,6 +184,113 @@ export async function predictShortlist(return { ...(result as SystemOneResult), shortlist: meta };}+/**+ * Reduce large choices through balanced elimination rounds, then predict all questions.+ * Each round batches every group's question in one call. Metadata lists the finalists;+ * probabilities, confidence, usage and other result fields come from the final call only.+ */+export async function predictTournament(+ agent: unknown,+ state: unknown,+ questions: Record<string, QuestionDef>,+ groupSize: number = DEFAULT_TOURNAMENT_GROUP,+ predictKwargs: Record<string, unknown> = {},+): Promise<SystemOneResult & { tournament: Record<string, TournamentMeta> }> {+ if (typeof questions !== "object" || questions === null || Array.isArray(questions) ||+ (Object.getPrototypeOf(questions) !== null && Object.getPrototypeOf(questions) !== Object.prototype)) {+ throw new TypeError("questions must be a dict of question id -> definition");+ }+ if (!Number.isSafeInteger(groupSize) || groupSize < 2) {+ throw new Error(`groupSize must be a safe integer >= 2, got ${JSON.stringify(groupSize)}`);+ }+ const meta = new Map<string, TournamentMeta>();+ for (const [qid, qdef] of Object.entries(questions)) {+ if (typeof qdef !== "object" || qdef === null || Array.isArray(qdef) || qdef["type"] !== "choice") continue;+ if (!("criteria" in qdef)) {+ throw new Error(`question ${JSON.stringify(qid)} is a choice but has no criteria`);+ }+ const criteria = qdef["criteria"];+ if (Array.isArray(criteria)) {+ for (const label of criteria) {+ if (label === null || label === undefined || typeof label === "object") {+ throw new Error(`question ${JSON.stringify(qid)}: choice labels must not be null, lists or dicts`);+ }+ }+ }+ const labels = criteriaItems(criteria).map(([label]) => label);+ meta.set(qid, { labels, n: labels.length, rounds: 0 });+ }+ const a = agent as Record<string, unknown>;+ const fn = a?.["predict"] ?? a?.["systemOne"] ?? a?.["system_one"];+ if (typeof fn !== "function") throw new TypeError("agent must provide predict or system_one");+ const extra = Object.keys(predictKwargs ?? {}).length ? [predictKwargs] : [];+ const predict = async (current: Record<string, QuestionDef>): Promise<SystemOneResult> => {+ const result = await (fn as (...args: unknown[]) => unknown).call(agent, state, current, ...extra);+ if (typeof result !== "object" || result === null || Array.isArray(result)) {+ throw new TypeError("predict/system_one must return a dict");+ }+ return result as SystemOneResult;+ };+ const warned = new Set<string>();+ const cut = (qid: string, labels: string[]): QuestionDef => {+ const original = questions[qid];+ const narrowed: QuestionDef = { ...original, criteria: subsetCriteria(original["criteria"], labels) };+ if (Object.hasOwn(narrowed, "option_order")) {+ delete narrowed["option_order"];+ if (!warned.has(qid)) {+ warned.add(qid);+ console.warn(+ `laya.shortlist: question ${JSON.stringify(qid)}: 'option_order' was dropped because the label set was narrowed to ${labels.length}; the order no longer fits the kept options.`,+ );+ }+ }+ return narrowed;+ };++ while (true) {+ const groups: { qid: string; labels: string[] }[] = [];+ for (const [qid, entry] of meta) {+ const n = entry.labels.length;+ const parts = Math.ceil(n / groupSize);+ if (parts <= 1) continue;+ for (let i = 0; i < parts; i++) {+ groups.push({ qid, labels: entry.labels.slice(Math.floor(i * n / parts), Math.floor((i + 1) * n / parts)) });+ }+ }+ if (groups.length === 0) break;+ const roundQuestions = Object.fromEntries(groups.map(({ qid, labels }, i) => [String(i), cut(qid, labels)]));+ const result = await predict(roundQuestions);+ const answers = result["answers"];+ if (typeof answers !== "object" || answers === null || Array.isArray(answers)) {+ throw new TypeError("tournament round must return an answers dict");+ }+ const winners = new Map<string, string[]>();+ for (const [i, { qid, labels }] of groups.entries()) {+ const answer = Object.hasOwn(answers, String(i)) ? (answers as Record<string, unknown>)[String(i)] : undefined;+ const choice = typeof answer === "object" && answer !== null && !Array.isArray(answer) && Object.hasOwn(answer, "choice")+ ? (answer as Record<string, unknown>)["choice"] : undefined;+ if (typeof choice !== "string" || !labels.includes(choice)) {+ throw new Error(`tournament group ${i} for question ${JSON.stringify(qid)} must return a choice from its criteria`);+ }+ const kept = winners.get(qid) ?? [];+ kept.push(choice);+ winners.set(qid, kept);+ }+ for (const [qid, labels] of winners) {+ const entry = meta.get(qid)!;+ entry.labels = labels;+ entry.rounds++;+ }+ }++ const finalQuestions = Object.fromEntries(Object.entries(questions).map(([qid, qdef]) => {+ const entry = meta.get(qid);+ return [qid, entry?.rounds ? cut(qid, entry.labels) : qdef];+ }));+ const result = await predict(finalQuestions);+ return { ...result, tournament: Object.fromEntries(meta) };+}+/** Mean-pool the agent's provider encoder. Throws honestly without encoder access. */export function embedFnFromAgent(agent: unknown, maxLength = 512, batchSize = 32): EmbedFn {if (!Number.isInteger(maxLength) || (maxLength as number) < 1) {diff --git a/laya-ts/tests/package-e2e.mjs b/laya-ts/tests/package-e2e.mjsindex ce646d96..ab60201a 100644--- a/laya-ts/tests/package-e2e.mjs+++ b/laya-ts/tests/package-e2e.mjs@@ -107,7 +107,7 @@ test("the packed module installs and works in an external ESM TypeScript projectawait writeFile(join(consumerDir, "runtime.mjs"),`import assert from "node:assert/strict";-import { Agent, VERSION } from "laya-ts";+import { Agent, VERSION, predictTournament, DEFAULT_TOURNAMENT_GROUP } from "laya-ts";let encoderCalls = 0;let headCalls = 0;@@ -136,6 +136,12 @@ assert.equal(result.answers.intent.probabilities.refund > 0.5, true);assert.equal(result.usage.output_tokens, 0);assert.equal(encoderCalls, 1);assert.equal(headCalls, 1);+assert.equal(DEFAULT_TOURNAMENT_GROUP, 16);+const tournament = await predictTournament(agent, "charged twice", {+ intent: { type: "choice", instructions: "Pick a route", criteria: ["refund", "other"] },+});+assert.equal(tournament.answers.intent.choice, "refund");+assert.deepEqual(tournament.tournament.intent, { labels: ["refund", "other"], n: 2, rounds: 0 });`,);await run(process.execPath, ["runtime.mjs"], consumerDir);@@ -145,9 +151,12 @@ assert.equal(headCalls, 1);`import {Agent,VERSION,+ predictTournament,+ DEFAULT_TOURNAMENT_GROUP,type QuestionDef,type SessionProvider,type SystemOneResult,+ type TournamentMeta,} from "laya-ts";const provider: SessionProvider = {@@ -175,6 +184,9 @@ const prediction: Promise<SystemOneResult> = new Agent({ provider }).predict(void VERSION;void prediction;+const tournament: Promise<SystemOneResult & { tournament: Record<string, TournamentMeta> }> =+ predictTournament(new Agent({ provider }), "charged twice", questions, DEFAULT_TOURNAMENT_GROUP);+void tournament;`,);await writeFile(diff --git a/laya-ts/tests/tournament.test.ts b/laya-ts/tests/tournament.test.tsnew file mode 100644index 00000000..1cc6529c--- /dev/null+++ b/laya-ts/tests/tournament.test.ts@@ -0,0 +1,302 @@+import { readFileSync } from "node:fs";+import { describe, expect, it, vi } from "vitest";+import { Agent, Router, DEFAULT_TOURNAMENT_GROUP, predictTournament } from "../src/index.js";+import type { QuestionDef, SessionProvider } from "../src/index.js";++type Questions = Record<string, QuestionDef>;+const labels = (q: QuestionDef): string[] => Array.isArray(q.criteria)+ ? q.criteria.map(String) : Object.keys(q.criteria as object);+const choice = (criteria: unknown): QuestionDef => ({ type: "choice", instructions: "Choose a route.", criteria });+const names = (n: number): string[] => Array.from({ length: n }, (_, i) => `label-${i}`);++class Ranker {+ calls: { state: unknown; questions: Questions; opts: Record<string, unknown> | undefined }[] = [];+ returned: Record<string, any>[] = [];++ constructor(readonly better: (a: string, b: string) => boolean = () => false) {}++ async predict(state: unknown, questions: Questions, opts?: Record<string, unknown>) {+ this.calls.push({ state, questions, opts });+ const answers = Object.fromEntries(Object.entries(questions).map(([qid, q]) => {+ if (q.type !== "choice") return [qid, { type: q.type, score: 0.5, noul: 0.5 }];+ const options = labels(q);+ const winner = options.reduce((best, next) => this.better(next, best) ? next : best);+ return [qid, { type: "choice", choice: winner, probabilities: { [winner]: 1 }, confidence: 1 }];+ }));+ const result = {+ model: "fixture", answers, usage: { input_tokens: this.calls.length, output_tokens: 0 },+ routing: { model: "fixture", reason: "fixture route" }, trace: this.calls.length,+ };+ this.returned.push(result);+ return result;+ }+}++// Reuse the existing Python-generated oracle, not another copy of the grouping algorithm.+// No Java toolchain or checkpoint is needed to read these recorded calls.+const golden = JSON.parse(readFileSync(new URL("../../laya-java/fixtures/tournament.json", import.meta.url), "utf8")) as {+ default_group_size: number;+ rank_probe: Record<string, string>;+ splits: { group_size: number; n: number; rounds: number[][]; finalists: string[]; choice: string }[];+};+function rank(label: string): bigint {+ let value = 0xcbf29ce484222325n;+ for (const byte of new TextEncoder().encode(`rank:${label}`)) {+ value = BigInt.asUintN(64, (value ^ BigInt(byte)) * 0x100000001b3n);+ }+ return value;+}++describe("tournament decisions", () => {+ it("uses the reference default and the independently recorded rank function", () => {+ expect(DEFAULT_TOURNAMENT_GROUP).toBe(16);+ expect(DEFAULT_TOURNAMENT_GROUP).toBe(golden.default_group_size);+ expect(golden.splits.length).toBeGreaterThanOrEqual(100);+ for (const [label, value] of Object.entries(golden.rank_probe)) expect(String(rank(label))).toBe(value);+ });++ it.each(golden.splits)("matches Python's recorded rounds for $n labels / groups of $group_size", async (row) => {+ const options = Array.from({ length: row.n }, (_, i) => `l${String(i).padStart(4, "0")}`);+ const runner = new Ranker((a, b) => rank(a) < rank(b));+ const result = await predictTournament(runner, "state", { q: choice(options) }, row.group_size);+ expect(runner.calls.slice(0, -1).map((c) => Object.values(c.questions).map((q) => labels(q).length)))+ .toEqual(row.rounds);+ expect(result.tournament.q).toEqual({ labels: row.finalists, n: row.n, rounds: row.rounds.length });+ expect(result.answers.q).toMatchObject({ choice: row.choice });+ });++ it.each([77, 257])("a consistently best label survives from every position among %i options", async (n) => {+ const options = names(n);+ for (const best of [options[0], options[Math.floor(n / 2)], options[n - 1]]) {+ const runner = new Ranker((a) => a === best);+ const result = await predictTournament(runner, "state", { q: choice(options) });+ expect(result.answers.q).toMatchObject({ choice: best });+ expect(runner.calls).toHaveLength(n === 77 ? 2 : 3);+ if (n === 77) {+ expect(Object.values(runner.calls[0].questions).map((q) => labels(q).length)).toEqual([15, 15, 16, 15, 16]);+ }+ const firstRound = Object.values(runner.calls[0].questions).flatMap(labels);+ expect(firstRound).toEqual(options);+ expect(new Set(firstRound).size).toBe(n);+ }+ });++ it("combines differently sized questions into rounds and preserves the final result", async () => {+ const big = choice(Object.fromEntries(names(20).map((label) => [label, { description: label }])));+ const medium = choice(["m0", "m1", "m2", "m3", "m4"]);+ const small = choice(["x", "y"]);+ const score: QuestionDef = { type: "score", instructions: "Rate urgency.", criteria: ["low", "high"] };+ const questions = { big, medium, small, score };+ const original = structuredClone(questions);+ const state = { text: "request", turns: ["original"] };+ const onPredictStart = vi.fn();+ const opts = { model: "english", lang: "de", hooks: [{ onPredictStart }], minConfidence: 0.8, headMaxLen: 160 };+ const runner = new Ranker();+ const result = await predictTournament(runner, state, questions, 4, opts);+ expect(runner.calls).toHaveLength(3);+ expect(runner.calls.map((c) => Object.keys(c.questions).length)).toEqual([7, 2, 4]);+ for (const call of runner.calls) {+ expect(call.state).toBe(state);+ expect(call.opts).toEqual(opts);+ expect(call.opts?.hooks).toBe(opts.hooks);+ }+ expect(result.tournament.big).toEqual({ labels: ["label-0", "label-8"], n: 20, rounds: 2 });+ expect(result.tournament.medium).toEqual({ labels: ["m0", "m2"], n: 5, rounds: 1 });+ expect(result.tournament.small).toEqual({ labels: ["x", "y"], n: 2, rounds: 0 });+ const final = runner.calls.at(-1)!.questions;+ expect(final.small).toBe(small);+ expect(final.score).toBe(score);+ expect(final.big.criteria).toEqual({ "label-0": { description: "label-0" }, "label-8": { description: "label-8" } });+ expect(questions).toEqual(original);+ const returned = runner.returned.at(-1)!;+ expect(result).toEqual({ ...returned, tournament: result.tournament });+ expect(result.answers).toBe(returned.answers);+ expect(result.usage).toBe(returned.usage);+ expect(returned).not.toHaveProperty("tournament");+ });++ it.each([{}, { small: choice(["x", "y"]), flag: { type: "noul", instructions: "Is it urgent?" } }])(+ "uses just one final call for an empty or already small request", async (questions) => {+ const runner = new Ranker();+ await predictTournament(runner, "state", questions as Questions);+ expect(runner.calls).toHaveLength(1);+ expect(runner.calls[0].questions).toEqual(questions);+ },+ );++ it("awaits a round before asking finalists, and preserves rejection identity", async () => {+ let resolve!: (value: unknown) => void;+ const blocked = new Promise((done) => { resolve = done; });+ const predict = vi.fn().mockReturnValueOnce(blocked).mockResolvedValue({ answers: {}, usage: {} });+ const pending = predictTournament({ predict }, "state", { q: choice(names(5)) }, 4);+ await Promise.resolve();+ expect(predict).toHaveBeenCalledTimes(1);+ resolve({ answers: { "0": { type: "choice", choice: "label-0" }, "1": { type: "choice", choice: "label-2" } } });+ await pending;+ expect(predict).toHaveBeenCalledTimes(2);+ const error = new Error("provider stopped");+ const reject = vi.fn().mockRejectedValue(error);+ await expect(predictTournament({ predict: reject }, "state", { q: choice(names(20)) }, 2)).rejects.toBe(error);+ expect(reject).toHaveBeenCalledTimes(1);+ });++ it.each(["systemOne", "system_one"])("supports the %s fallback with its receiver", async (method) => {+ const runner = new Ranker();+ const adapter = { owner: runner, [method]: function (this: { owner: Ranker }, ...args: Parameters<Ranker["predict"]>) {+ return this.owner.predict(...args);+ } };+ await predictTournament(adapter, "state", { q: choice(names(5)) }, 4);+ expect(runner.calls).toHaveLength(2);+ });++ it("keeps special question IDs and labels as own entries", async () => {+ const options = ["__proto__", "constructor", "toString", "safe", "last"];+ const questions = Object.fromEntries(["__proto__", "constructor", "toString"].map((qid) => [qid, choice(options)]));+ const runner = new Ranker();+ const result = await predictTournament(runner, "state", questions, 4);+ expect(Object.keys(result.tournament)).toEqual(Object.keys(questions));+ for (const qid of Object.keys(questions)) {+ expect(Object.hasOwn(result.tournament, qid)).toBe(true);+ expect(result.tournament[qid].labels).toEqual(["__proto__", "toString"]);+ expect(Object.hasOwn(runner.calls.at(-1)!.questions, qid)).toBe(true);+ }+ const dictRunner = new Ranker();+ await predictTournament(dictRunner, "state", { q: choice(Object.fromEntries(options.map((label) => [label, label]))) }, 4);+ expect(Object.keys(dictRunner.calls.at(-1)!.questions.q.criteria as object)).toEqual(["__proto__", "toString"]);+ expect(({} as Record<string, unknown>).polluted).toBeUndefined();+ });++ it("uses native object enumeration for numeric keys and preserves array labels", async () => {+ const criteria = Object.fromEntries(["10", "2", "1", "z", "a"].map((k) => [k, `about ${k}`]));+ const runner = new Ranker();+ await predictTournament(runner, "state", { q: choice(criteria) }, 4);+ expect(Object.values(runner.calls[0].questions).flatMap(labels)).toEqual(["1", "2", "10", "z", "a"]);+ const arrays = new Ranker();+ const result = await predictTournament(arrays, "state", { q: choice([3, false, "x", 9, "z"]) }, 4);+ expect(result.tournament.q.labels).toEqual(["3", "x"]);+ });++ it("drops narrowed option_order once per question, preserving passthrough and the caller", async () => {+ const warn = vi.spyOn(console, "warn").mockImplementation(() => {});+ try {+ const big = { ...choice(names(20)), option_order: names(20).map((_, i) => 19 - i) };+ const small = { ...choice(["x", "y"]), option_order: [1, 0] };+ const before = structuredClone({ big, small });+ const runner = new Ranker();+ await predictTournament(runner, "state", { big, small }, 4);+ expect(warn).toHaveBeenCalledTimes(1);+ expect(warn.mock.calls[0].join(" ")).toContain("big");+ expect(warn.mock.calls[0].join(" ")).toContain("option_order");+ for (const call of runner.calls.slice(0, -1)) {+ for (const q of Object.values(call.questions)) expect(q).not.toHaveProperty("option_order");+ }+ expect(runner.calls.at(-1)!.questions.big).not.toHaveProperty("option_order");+ expect(runner.calls.at(-1)!.questions.small).toBe(small);+ expect({ big, small }).toEqual(before);+ } finally {+ warn.mockRestore();+ }+ });++ it.each([1, 0, -1, 2.5, NaN, Infinity, Number.MAX_SAFE_INTEGER + 1, true, "16", null])("rejects invalid group size %s before inference", async (groupSize) => {+ const predict = vi.fn();+ await expect(predictTournament({ predict }, "state", {}, groupSize as number)).rejects.toThrow();+ expect(predict).not.toHaveBeenCalled();+ });++ it.each([undefined, null, [], {}, "a", ["a", "a"], [1, "1"]].map((criteria) => ({ criteria })))("rejects invalid criteria $criteria before any group", async ({ criteria }) => {+ const predict = vi.fn();+ await expect(predictTournament({ predict }, "state", { valid: choice(names(20)), bad: choice(criteria) }, 4)).rejects.toThrow();+ expect(predict).not.toHaveBeenCalled();+ });++ it.each([null, [], "questions", 10, new Map(), new Set()].map((questions) => ({ questions })))("rejects values that are not question dictionaries: $questions", async ({ questions }) => {+ const predict = vi.fn();+ await expect(predictTournament({ predict }, "state", questions as Questions)).rejects.toThrow();+ expect(predict).not.toHaveBeenCalled();+ });++ it.each([[null, "b", "c"], [undefined, "b", "c"], [["nested"], "b", "c"], [{ route: "bad" }, "b", "c"]].map((criteria) => ({ criteria })))(+ "does not turn an invalid label into a valid string while narrowing $criteria", async ({ criteria }) => {+ const predict = vi.fn();+ await expect(predictTournament({ predict }, "state", { q: choice(criteria) }, 2)).rejects.toThrow();+ expect(predict).not.toHaveBeenCalled();+ },+ );++ it("accepts null-prototype dictionaries without losing their entries", async () => {+ const questions = Object.assign(Object.create(null), { q: choice(names(5)) });+ const runner = new Ranker();+ const result = await predictTournament(runner, "state", questions, 4);+ expect(result.tournament.q).toEqual({ labels: ["label-0", "label-2"], n: 5, rounds: 1 });+ });++ it.each([null, [], "not a result"].map((result) => ({ result })))("rejects final results that are not dictionaries: $result", async ({ result }) => {+ const predict = vi.fn().mockResolvedValue(result);+ await expect(predictTournament({ predict }, "state", { q: choice(["a"]) })).rejects.toThrow();+ expect(predict).toHaveBeenCalledTimes(1);+ });++ it.each([+ null, [], {}, { answers: {} }, { answers: { "0": { type: "choice", choice: "label-0" } } },+ { answers: { "0": { type: "choice", choice: "outside" }, "1": { type: "choice", choice: "label-2" } } },+ { answers: { "0": { type: "choice", choice: "label-2" }, "1": { type: "choice", choice: "label-2" } } },+ { answers: Object.create({ "0": { choice: "label-0" }, "1": { choice: "label-2" } }) },+ { answers: { "0": Object.create({ choice: "label-0" }), "1": { choice: "label-2" } } },+ ].map((output) => ({ output })))("stops instead of advancing malformed or foreign group winners $output", async ({ output }) => {+ const predict = vi.fn().mockResolvedValue(output);+ await expect(predictTournament({ predict }, "state", { q: choice(names(5)) }, 4)).rejects.toThrow();+ expect(predict).toHaveBeenCalledTimes(1);+ });+});++function realAgent() {+ const batches: number[] = [];+ const provider: SessionProvider = {+ async runEncoder(batch) {+ batches.push(batch.qtype.length);+ return { lastHidden: batch.inputIds.map((row) => row.map(() => [0])) };+ },+ async runHead(_hidden, batch) {+ return {+ logits: batch.markerMask.map((mask) => mask.map((_, i) => i === 0 ? 2 : 0)),+ act: batch.qtype.map(() => [3, 0]),+ };+ },+ };+ return { agent: new Agent({ provider }), batches };+}++describe("tournament runtime integration", () => {+ it("executes each round as one real Agent provider batch and exposes final-only probabilities", async () => {+ const { agent, batches } = realAgent();+ const starts = vi.fn();+ const result = await predictTournament(agent, "request", { q: choice(names(20)) }, 4, {+ onPredictStart: starts, minConfidence: 1,+ });+ expect(batches).toEqual([5, 2, 1]);+ expect(starts).toHaveBeenCalledTimes(3);+ expect(result.answers.q.type).toBe("choice");+ if (result.answers.q.type !== "choice") throw new Error("expected choice");+ expect(Object.keys(result.answers.q.probabilities)).toEqual(result.tournament.q.labels);+ expect(result.answers.q.low_confidence).toBe(true);+ expect(result.usage.input_tokens).toBeGreaterThan(0);+ expect(result.model).toBe("laya-rl-agent");+ });++ it("preserves Router routing and runs explicit model options and hooks in every round", async () => {+ const { agent, batches } = realAgent();+ const router = new Router();+ router.attach("english", agent);+ const routes: unknown[] = [];+ const hooks = { onPredictStart(ctx: { decision?: unknown }) { routes.push(ctx.decision); } };+ const result = await predictTournament(router, "退款", { q: choice(names(5)) }, 4, {+ model: "english", hooks, minConfidence: 1,+ });+ expect(batches).toEqual([2, 1]);+ expect(routes).toHaveLength(2);+ for (const route of routes) expect(route).toMatchObject({ model: "english" });+ expect(result).toHaveProperty("routing.model", "english");+ expect(result.answers.q.low_confidence).toBe(true);+ });+});diff --git a/laya-ts/tests/types/tournament.ts b/laya-ts/tests/types/tournament.tsnew file mode 100644index 00000000..56a9bf55--- /dev/null+++ b/laya-ts/tests/types/tournament.ts@@ -0,0 +1,30 @@+import {+ Agent, Router, predictTournament, DEFAULT_TOURNAMENT_GROUP,+ type QuestionDef, type SystemOneResult, type TournamentMeta,+} from "../../src/index.js";++declare const agent: Agent;+declare const router: Router;+declare const questions: Record<string, QuestionDef>;++const count: number = DEFAULT_TOURNAMENT_GROUP;+const agentResult: Promise<SystemOneResult & { tournament: Record<string, TournamentMeta> }> =+ predictTournament(agent, "request", questions);+const routerResult: Promise<SystemOneResult & { tournament: Record<string, TournamentMeta> }> =+ predictTournament(router, "request", questions, 8, { model: "english", lang: "en" });++async function inspect() {+ const result = await agentResult;+ const finalists: string[] = result.tournament.route.labels;+ const originalCount: number = result.tournament.route.n;+ const rounds: number = result.tournament.route.rounds;+ // @ts-expect-error group size is numeric, not a numeric string+ predictTournament(agent, "request", questions, "16");+ // @ts-expect-error finalist labels are strings+ const invalid: number[] = result.tournament.route.labels;+ return { finalists, originalCount, rounds, invalid };+}++void count;+void routerResult;+void inspect;diff --git a/tests/test_hooks_api.py b/tests/test_hooks_api.pyindex 9b4b0c6d..edc68aad 100644--- a/tests/test_hooks_api.py+++ b/tests/test_hooks_api.py@@ -178,8 +178,15 @@ for param in ("max_len", "head_max_len"):check_param("shortlist_choice", laya.shortlist_choice, "return_scores", False,inspect.Parameter.KEYWORD_ONLY)-# predict_tournament splits a choice into groups of 16 labels unless the caller picks a size-check_param("predict_tournament", laya.predict_tournament, "group_size", 16)+# The TypeScript helper follows this positional contract; both default to groups of 16.+check_param("predict_tournament", laya.predict_tournament, "group_size", 16,+ inspect.Parameter.POSITIONAL_OR_KEYWORD)+check("predict_tournament/positional prefix",+ [p.name for p in sig(laya.predict_tournament).values()+ if p.kind == inspect.Parameter.POSITIONAL_OR_KEYWORD],+ ["agent", "state", "questions", "group_size"])+check_param("predict_tournament", laya.predict_tournament, "predict_kwargs", inspect.Parameter.empty,+ inspect.Parameter.VAR_KEYWORD)# route() and route_batch() take per-call hooks so a hook can pin a checkpoint for one callfor label, fn in (("Router.route", Router.route),