Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
research(step4): reproduce typed-decisions through laya-train
Bruce-Yii2a16e8033169
diff --git a/research/scripts/step4_benchmark/README.md b/research/scripts/step4_benchmark/README.mdnew file mode 100644index 00000000..ebcc2f92--- /dev/null+++ b/research/scripts/step4_benchmark/README.md@@ -0,0 +1,149 @@+# Step 4 benchmark: reproduce typed-decisions through `laya-train`++Harness for #887 Step 4. Two evidence surfaces, kept separate:++1. **Typed-decisions reproduction** — the fine-tune is reproduced through the shipped+ `laya-train` CLI and scored on the official test split: **accuracy, ECE, Brier**. This is the+ surface that verifies acceptance criterion 1.+2. **20-option order stability** — measured on **`massive_intent.en` at 20 options**, not on+ typed-decisions (whose `choice` questions are 4-option). The original issue names that surface+ explicitly; do not substitute a "20-option typed-decisions" run.++The default loss is still `rlcd`. Whether `soft-ce` should replace it is decided by the Gate D+matrix below, not by a single seed.++## Files++| File | Purpose |+|---|---|+| `prepare_data.py` | `LocalLLaMA/typed-decisions` → `train.jsonl` / `test.jsonl` / `test_meta.json` |+| `run_benchmark.py` | Runs `laya-train` for a gate's loss × seed matrix; writes a per-run manifest |+| `evaluate.py` | Accuracy / ECE / Brier on the test split, under shipped and raw temperatures |+| `metrics.py` | The notebook's metric definitions (shared by `evaluate.py` and the smoke test) |+| `order_flip.py` | Inference-only 20-option flip rate on `massive_intent.en` |+| `smoke_test.py` | CPU, weight-free: metric plumbing + CLI wiring + tiny end-to-end |++## Exact benchmark contract++Pinned from the published notebook (`notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb`)+and #887's thread. Confirm against the base checkpoint config at run time (read once, record it).++| | value | source |+|---|---|---|+| dataset | `LocalLLaMA/typed-decisions`, config `all` | notebook |+| train split | `train` — 1,200 cases / 6,000 decisions | notebook |+| test split | `test` — 400 cases / 2,000 decisions | notebook |+| base checkpoint | `convaiinnovations/laya` | notebook |+| loss (reproduction) | `rlcd` (`laya-train` default) | #887 |+| epochs | 4 | notebook |+| micro-batch / grad-accum | 8 / 8 — effective batch 64 single-device (notebook: 8 × 2 GPUs × 4) | notebook, `TrainConfig` |+| learning rates | encoder 2.5e-5, head 1e-4, AdamW cosine | notebook |+| `max_len` / `head_max_len` | 1024 / 256 | notebook, `docs/finetune.md` |+| training seed | 0 | #887 |+| calibration slice | held out inside `laya-train`: `calib_frac` 0.1 capped at 400, seed 20260922 | `TrainConfig`, notebook |+| mixed precision | fp16 autocast on CUDA; gradient checkpointing follows it | notebook, `TrainConfig` |++### Metric definitions (notebook cell 14)++- **accuracy** — top option vs gold label, over every decision (choice + noul + score).+- **ECE** — `laya.common.ece_score` of the top probability against correctness, 15 equal-width+ bins, over every decision.+- **Brier** — mean over **choice and noul** decisions of `sum((p - gold)**2)`; the notebook does+ not append score decisions to this mean, so a score change never moves Brier. Kept as-is so the+ number compares with the reference.+- **soft accuracy** — mean over **choice and noul** of `sum(p * gold)`.++### Reference numbers to compare against++From #887 (same split and seed, notebooks linked in-thread):++| run | accuracy | ECE | Brier |+|---|---|---|---|+| base `laya` | 0.362 | 0.174 | 0.329 |+| `loss="rlcd"` (notebook objective) | 0.7715 | 0.150 | 0.050 |+| `loss="soft-ce"` | 0.790 | 0.157 | 0.047 |+| published `laya-typed-decisions` | 0.766 | 0.213 | 0.061 |++Accuracy is the cleanest training-reproduction signal; ECE and Brier must name the calibration+path that produced them (next section).++### Calibration is not byte-comparable across contracts++`laya.train` fits with `laya.calibrate.fit_temperature_map` — per type, plus per option-count+bucket above a floor — under the runtime clamp `[TEMP_MIN, TEMP_MAX] = [0.5, 5.0]`. The old+notebook fitted one LBFGS temperature per type under `[0.1, 10]`. So `evaluate.py` reports ECE and+Brier under **both** paths (`--temperatures shipped` uses the checkpoint's own fit; `--temperatures+raw` uses 1.0 everywhere). An ECE difference explained by the calibration-contract change is not a+training non-reproduction; say which path produced each number.++## Gates++### Gate R — reproduction++One run: `rlcd`, seed 0. Passes when accuracy lands on the published checkpoint / the maintainer's+target within single-seed noise, and ECE/Brier are reported with their calibration path. Nothing+about the default loss is concluded from Gate R.++```bash+python research/scripts/step4_benchmark/prepare_data.py --out-dir data/typed_decisions+python research/scripts/step4_benchmark/run_benchmark.py --gate R \+ --data data/typed_decisions/train.jsonl --out-dir runs/step4 --device cuda+python research/scripts/step4_benchmark/evaluate.py \+ --checkpoint runs/step4/rlcd_seed0 \+ --test-meta data/typed_decisions/test_meta.json --device cuda+```++### Gate D — default-loss decision++`rlcd` vs `soft-ce` over more than one seed. The existing soft-ce win is single-seed, so it is not+enough to recommend changing a shipped default. Minimal matrix (seeds 0/1/2, 6 runs):++```bash+python research/scripts/step4_benchmark/run_benchmark.py --gate D \+ --data data/typed_decisions/train.jsonl --out-dir runs/step4 --device cuda+```++Summarise paired per-decision outcomes across seeds, not just the two means.++### 20-option order stability++`massive_intent.en`, 20 options (gold + 19 distractors, seed 13, matching+`research/scripts/build_benchmark_nb.py`). GuilhermeFusari's seed-0/1 numbers from `laya.train`+already cover this surface; reference them unless a reviewer wants the number reproduced on a+specific checkpoint, in which case run inference-only:++```bash+python research/scripts/step4_benchmark/order_flip.py \+ --checkpoint <checkpoint> --per-lang 300 --out flip.json+```++This does not train. Do not start a second MASSIVE training campaign by inertia.++## Worked example / smoke++`smoke_test.py` runs on CPU with a tiny local checkpoint and a tiny typed-decisions-shaped JSONL.+It proves the metric definitions against hand-computed values, the CLI command, and the loop from+JSONL through `laya-train` to `evaluate.py`. It downloads nothing and trains nothing large.++```bash+python research/scripts/step4_benchmark/smoke_test.py+```++## Runtime (keep the three apart)++- **Reference (notebook, 2×T4):** ~4–6 minutes for the demo's 6,000 decisions.+- **Guilherme measured (one T4):** ~48–54 minutes per run — but that is the **MASSIVE 20-option**+ experiment, not typed-decisions. Do not carry it over.+- **Our measured:** fill in from `manifest_*.json` (`runs[].elapsed_seconds`) after a real run.++Single-device `laya-train` uses micro-batch 8 × grad-accum 8 = effective 64; the notebook used+8 × 2 GPUs × 4 = effective 64. Same effective batch, different step count (single device does+~4% more optimizer steps on the same data).++## Constraints++- Does not modify `laya/train.py`, `laya/train_cli.py` or `tests/test_train.py` (open PRs+ #967/#968/#982).+- Uses `laya-train` as a subprocess, so it does not depend on training internals.+- Reuses `laya.common.ece_score` and the notebook's own metric definitions rather than adding a+ second evaluation framework.diff --git a/research/scripts/step4_benchmark/evaluate.py b/research/scripts/step4_benchmark/evaluate.pynew file mode 100644index 00000000..a5fa9b67--- /dev/null+++ b/research/scripts/step4_benchmark/evaluate.py@@ -0,0 +1,190 @@+"""Evaluate a fine-tuned checkpoint on the typed-decisions test split.++Mirrors cell 14 of the published notebook, restricted to the three metrics the Step-4 contract+names: **accuracy**, **ECE** and **Brier**. The 20-option order-flip surface is deliberately not+here: it belongs to `massive_intent.en`, not typed-decisions (see README).++Calibration is not byte-comparable across contracts. `laya.train` fits temperatures with+`fit_temperature_map` (per type, and per option-count bucket above a floor) under the runtime+clamp `[0.5, 5.0]`; the old notebook fitted one LBFGS temperature per type under `[0.1, 10]`.+So ECE and Brier are reported under **both** paths:++- ``--temperatures shipped`` (default) — the checkpoint's own fitted `temperature` and+ `temperature_by_options`, which is what the notebook's number also used;+- ``--temperatures raw`` — temperature 1.0 everywhere, isolating the training from the+ calibration contract.++Accuracy is unaffected by temperature (it only rescales the logits), so it is the clean+reproduction signal.++Usage:+ python evaluate.py --checkpoint runs/step4/rlcd-seed0 \+ --test-meta data/typed_decisions/test_meta.json \+ --out runs/step4/rlcd-seed0/eval_shipped.json+"""+from __future__ import annotations++import argparse+import json+import os+import sys+from typing import Any, Dict, List++sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))+from metrics import ( # noqa: E402+ accuracy,+ brier_score,+ expected_calibration_error,+ soft_accuracy,+)+++def _normalise(values: List[float]) -> List[float]:+ total = sum(values)+ if total > 0:+ return [v / total for v in values]+ return values+++def evaluate_checkpoint(checkpoint_dir: str, test_meta: List[Dict[str, Any]],+ device: str = "auto", temperatures: str = "shipped") -> Dict[str, Any]:+ import laya++ agent = laya.load(checkpoint_dir, device=device)+ if temperatures == "raw":+ agent.temperature = [1.0, 1.0, 1.0]+ agent.temperature_by_options = {}+ elif temperatures != "shipped":+ raise ValueError("temperatures must be 'shipped' or 'raw', got %r" % (temperatures,))++ corrects: List[float] = []+ confidences: List[float] = []+ briers: List[float] = []+ softs: List[float] = []+ per_type: Dict[str, List[float]] = {"choice": [], "noul": [], "score": []}+ score_maes: List[float] = []+ within_one: List[float] = []++ for i, case in enumerate(test_meta):+ state = case["state"]+ questions = case["questions"]+ gold = case["gold"]+ answers = agent.predict(state, questions)["answers"]++ for qid, qdef in questions.items():+ if qid not in gold:+ continue+ g = gold[qid]+ qtype = qdef["type"]+ p_ans = answers[qid]++ if qtype == "choice":+ keys = list(qdef["criteria"].keys())+ pred = p_ans["choice"]+ gold_label = str(g["label"])+ is_corr = float(pred == gold_label)++ p_probs = _normalise([p_ans["probabilities"].get(k, 1e-6) for k in keys])+ g_probs = _normalise([g["probabilities"].get(k, 1e-6) for k in keys])++ corrects.append(is_corr)+ confidences.append(max(p_probs))+ briers.append(brier_score(p_probs, g_probs))+ softs.append(soft_accuracy(p_probs, g_probs))+ per_type["choice"].append(is_corr)++ elif qtype == "noul":+ p_val = float(p_ans["noul"])+ g_val = float(g.get("noul", g.get("probabilities", {}).get("true", 0.5)))+ gold_label = str(g["label"]).lower()+ pred_label = "true" if p_val >= 0.5 else "false"+ is_corr = float(pred_label == gold_label)++ p_dist = [1.0 - p_val, p_val]+ g_dist = [1.0 - g_val, g_val]++ corrects.append(is_corr)+ confidences.append(max(p_val, 1.0 - p_val))+ briers.append(brier_score(p_dist, g_dist))+ softs.append(soft_accuracy(p_dist, g_dist))+ per_type["noul"].append(is_corr)++ elif qtype == "score":+ p_score = float(p_ans["score"])+ g_score = float(g.get("score", 0.0))+ score_maes.append(abs(p_score - g_score))+ within_one.append(float(abs(p_score - g_score) <= 1.0))++ n_levels = len(qdef.get("criteria", []))+ p_probs = [p_ans["probabilities"].get(str(i), 0.0) for i in range(n_levels)]+ if sum(p_probs) > 0:+ p_probs = _normalise(p_probs)+ p_lvl = int(max(range(len(p_probs)), key=lambda j: p_probs[j]))+ confidences.append(max(p_probs))+ else:+ p_lvl = int(round(p_score))+ confidences.append(0.5)++ g_lvl = int(g.get("label", int(round(g_score))))+ is_corr = float(p_lvl == g_lvl)+ corrects.append(is_corr)+ per_type["score"].append(is_corr)++ if (i + 1) % 100 == 0:+ print(f" {i + 1}/{len(test_meta)} cases", flush=True)++ result = {+ "checkpoint": checkpoint_dir,+ "temperatures": temperatures,+ "n_cases": len(test_meta),+ "n_decisions": len(corrects),+ "accuracy": round(accuracy(corrects), 4),+ "ece": round(expected_calibration_error(confidences, corrects), 4),+ "brier": round(float(sum(briers) / len(briers)) if briers else float("nan"), 4),+ "soft_accuracy": round(float(sum(softs) / len(softs)) if softs else float("nan"), 4),+ "score_mae": round(float(sum(score_maes) / len(score_maes)) if score_maes else float("nan"), 4),+ "within_one_level": round(float(sum(within_one) / len(within_one)) if within_one else float("nan"), 4),+ "per_type_accuracy": {k: round(accuracy(v), 4) for k, v in per_type.items() if v},+ }+ # `brier`/`soft_accuracy`/`ece` mix question types; record the counts they rest on.+ result["n_brier_decisions"] = len(briers)+ return result+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--checkpoint", required=True)+ parser.add_argument("--test-meta", required=True)+ parser.add_argument("--device", default="auto")+ parser.add_argument("--temperatures", choices=["shipped", "raw"], default="shipped")+ parser.add_argument("--out", default=None,+ help="Output JSON (default: <checkpoint>/eval_<temperatures>.json)")+ args = parser.parse_args()++ with open(args.test_meta, encoding="utf-8") as f:+ test_meta = json.load(f)+ print(f"loaded {len(test_meta)} test cases", flush=True)++ result = evaluate_checkpoint(args.checkpoint, test_meta, device=args.device,+ temperatures=args.temperatures)++ print("\n" + "=" * 60)+ print(f"evaluation: {args.checkpoint} ({args.temperatures} temperatures)")+ print("=" * 60)+ print(f" accuracy: {result['accuracy']:.4f}")+ print(f" ECE: {result['ece']:.4f}")+ print(f" Brier: {result['brier']:.4f}")+ print(f" soft acc: {result['soft_accuracy']:.4f}")+ print(f" per type: {result['per_type_accuracy']}")+ print("=" * 60)++ out_path = args.out or os.path.join(args.checkpoint, f"eval_{args.temperatures}.json")+ os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)+ with open(out_path, "w", encoding="utf-8") as f:+ json.dump(result, f, indent=2)+ print(f"wrote {out_path}")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/research/scripts/step4_benchmark/metrics.py b/research/scripts/step4_benchmark/metrics.pynew file mode 100644index 00000000..4871efc9--- /dev/null+++ b/research/scripts/step4_benchmark/metrics.py@@ -0,0 +1,50 @@+"""Metric definitions for the Step-4 benchmark, mirroring the published notebook.++Reference: cell 14 of `notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb`. The published+0.766 checkpoint and the maintainer's reference table are scored with these definitions, so a+reproduction has to use them too, not a differently-defined library metric:++- **accuracy** — top reported option against the gold label, over every decision (choice, score+ and noul).+- **ECE** — `laya.common.ece_score` of the top probability against correctness, 15 equal-width+ bins, over every decision.+- **brier** — mean over **choice and noul** decisions of `sum((p - gold)**2)`. The notebook does+ not append score decisions to this mean, so a score change never moves Brier; kept as-is so the+ number compares with the reference.+- **soft_accuracy** — mean over **choice and noul** of `sum(p * gold)`.++`gold` is the teacher distribution (`gold.probabilities`), not a one-hot label, so Brier and soft+accuracy measure the distribution the run was trained to match.+"""+from __future__ import annotations++from typing import Sequence++import numpy as np++from laya.common import ece_score+++def accuracy(correct: Sequence[float]) -> float:+ return float(np.mean(correct)) if len(correct) else float("nan")+++def expected_calibration_error(confidences: Sequence[float], correct: Sequence[float],+ bins: int = 15) -> float:+ """Thin wrapper over the repository's `common.ece_score`, as the notebook calls it."""+ if not len(confidences):+ return float("nan")+ return float(ece_score(np.asarray(confidences, dtype=float),+ np.asarray(correct, dtype=float), bins=bins))+++def brier_score(probs: Sequence[float], gold: Sequence[float]) -> float:+ p = np.asarray(probs, dtype=float)+ g = np.asarray(gold, dtype=float)+ return float(((p - g) ** 2).sum())+++def soft_accuracy(probs: Sequence[float], gold: Sequence[float]) -> float:+ p = np.asarray(probs, dtype=float)+ g = np.asarray(gold, dtype=float)+ return float((p * g).sum())diff --git a/research/scripts/step4_benchmark/prepare_data.py b/research/scripts/step4_benchmark/prepare_data.pynew file mode 100644index 00000000..be9a6f30--- /dev/null+++ b/research/scripts/step4_benchmark/prepare_data.py@@ -0,0 +1,96 @@+"""Prepare LocalLLaMA/typed-decisions for `laya-train` and for evaluation.++The benchmark's own rows are `{state, questions, gold}` with the teacher distribution per+question. That is exactly the schema `laya.train` reads, so the conversion is a parse, not a+reshape: the two JSON-string columns are decoded and the row is written back out as JSONL.++Splits:++* ``train.jsonl`` — the dataset's ``train`` split (1,200 cases / 6,000 decisions), the input to+ ``laya-train``. The calibration slice is held out *inside* ``laya-train`` at a fixed seed+ (``calib_frac`` 0.1, capped at 400, seed 20260922), matching the published notebook.+* ``test.jsonl`` / ``test_meta.json`` — the dataset's ``test`` split (400 cases / 2,000+ decisions), used by ``evaluate.py``. ``test_meta.json`` keeps ``id`` and ``workflow`` so+ per-workflow accuracy can be reported alongside the pooled number.++Usage:+ python prepare_data.py --out-dir data/typed_decisions+"""+from __future__ import annotations++import argparse+import json+import os+import sys+from typing import Any, Dict, List+++def _parse(value: Any) -> Any:+ if isinstance(value, str):+ return json.loads(value)+ return value+++def case_to_record(row: Dict[str, Any]) -> Dict[str, Any]:+ """One dataset row as a `laya-train` record (`{state, questions, gold}`)."""+ return {+ "state": _parse(row["state"]),+ "questions": _parse(row["questions"]),+ "gold": _parse(row["gold"]),+ }+++def load_split(split: str, dataset: str = "LocalLLaMA/typed-decisions") -> List[Dict[str, Any]]:+ try:+ from datasets import load_dataset+ except ImportError:+ print("ERROR: the datasets package is required (pip install datasets)", file=sys.stderr)+ sys.exit(1)+ ds = load_dataset(dataset, "all", split=split)+ return list(ds)+++def write_split(rows: List[Dict[str, Any]], out_dir: str, split: str) -> int:+ records = [case_to_record(r) for r in rows]+ path = os.path.join(out_dir, f"{split}.jsonl")+ with open(path, "w", encoding="utf-8") as f:+ for rec in records:+ f.write(json.dumps(rec, ensure_ascii=False) + "\n")+ return len(records)+++def write_test_meta(rows: List[Dict[str, Any]], out_dir: str) -> int:+ meta = []+ for r in rows:+ meta.append({+ "id": r.get("id", ""),+ "workflow": r.get("workflow", ""),+ "state": _parse(r["state"]),+ "questions": _parse(r["questions"]),+ "gold": _parse(r["gold"]),+ })+ path = os.path.join(out_dir, "test_meta.json")+ with open(path, "w", encoding="utf-8") as f:+ json.dump(meta, f, indent=2)+ return len(meta)+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--out-dir", default="data/typed_decisions")+ parser.add_argument("--dataset", default="LocalLLaMA/typed-decisions")+ args = parser.parse_args()++ os.makedirs(args.out_dir, exist_ok=True)+ for split in ("train", "test"):+ rows = load_split(split, args.dataset)+ n = write_split(rows, args.out_dir, split)+ print(f"{split}: {n} cases -> {os.path.join(args.out_dir, split + '.jsonl')}")+ if split == "test":+ m = write_test_meta(rows, args.out_dir)+ print(f"test meta: {m} cases -> {os.path.join(args.out_dir, 'test_meta.json')}")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/research/scripts/step4_benchmark/run_benchmark.py b/research/scripts/step4_benchmark/run_benchmark.pynew file mode 100644index 00000000..ca411700--- /dev/null+++ b/research/scripts/step4_benchmark/run_benchmark.py@@ -0,0 +1,150 @@+"""Run the Step-4 benchmark through the `laya-train` CLI.++Two gates, kept separate on purpose:++* **Gate R — reproduction.** Seed 0, the notebook's hyperparameters, `--loss rlcd`. It has to land+ on the published checkpoint / the maintainer's target within noise; that verifies acceptance+ criterion 1. Nothing else is claimed from one seed.+* **Gate D — default-loss decision.** `rlcd` vs `soft-ce` over more than one seed. The existing+ single-seed soft-ce win is not enough to recommend changing a shipped default, so Gate D is+ only designed and run after Gate R passes.++This calls the CLI as a subprocess: it does not import `laya.train`, so it cannot depend on the+training internals #967/#968/#982 are still changing.++Usage:+ # Gate R: one run+ python run_benchmark.py --gate R --data data/typed_decisions/train.jsonl \+ --out-dir runs/step4 --device cuda++ # Gate D: rlcd + soft-ce over seeds 0/1/2+ python run_benchmark.py --gate D --data data/typed_decisions/train.jsonl \+ --out-dir runs/step4 --device cuda+"""+from __future__ import annotations++import argparse+import json+import os+import shlex+import subprocess+import sys+import time+from typing import Any, Dict, List, Tuple++# The notebook's recipe, single-device form. 8 micro-batch x 2 GPUs x 4 accumulation = 64+# effective; on one device that is 8 x 8.+NOTEBOOK_HYPERPARAMS = {+ "epochs": 4,+ "micro_batch": 8,+ "grad_accum": 8,+ "encoder_lr": 2.5e-5,+ "head_lr": 1e-4,+ "max_len": 1024,+ "head_max_len": 256,+}++GATES: Dict[str, List[Tuple[str, int]]] = {+ "R": [("rlcd", 0)],+ "D": [("rlcd", 0), ("rlcd", 1), ("rlcd", 2),+ ("soft-ce", 0), ("soft-ce", 1), ("soft-ce", 2)],+}+++def build_command(data: str, out_dir: str, loss: str, seed: int, base: str, device: str,+ hyper: Dict[str, Any]) -> List[str]:+ return [+ sys.executable, "-m", "laya.train_cli",+ "--data", data,+ "--out", out_dir,+ "--base", base,+ "--loss", loss,+ "--seed", str(seed),+ "--epochs", str(hyper["epochs"]),+ "--micro-batch", str(hyper["micro_batch"]),+ "--grad-accum", str(hyper["grad_accum"]),+ "--encoder-lr", str(hyper["encoder_lr"]),+ "--head-lr", str(hyper["head_lr"]),+ "--max-len", str(hyper["max_len"]),+ "--head-max-len", str(hyper["head_max_len"]),+ "--device", device,+ ]+++def run_one(data: str, out_root: str, loss: str, seed: int, base: str, device: str,+ hyper: Dict[str, Any]) -> Dict[str, Any]:+ run_dir = os.path.join(out_root, f"{loss.replace('-', '_')}-seed{seed}")+ os.makedirs(run_dir, exist_ok=True)+ cmd = build_command(data, run_dir, loss, seed, base, device, hyper)+ print("\n" + "=" * 72)+ print(" ".join(shlex.quote(c) for c in cmd))+ print("=" * 72, flush=True)++ t0 = time.time()+ result = subprocess.run(cmd, text=True)+ elapsed = round(time.time() - t0, 1)+ if result.returncode != 0:+ raise SystemExit(f"laya-train failed for loss={loss} seed={seed} "+ f"(exit {result.returncode})")++ cfg_path = os.path.join(run_dir, "rl_agent_config.json")+ cfg: Dict[str, Any] = {}+ if os.path.exists(cfg_path):+ with open(cfg_path, encoding="utf-8") as f:+ cfg = json.load(f)++ return {+ "loss": loss,+ "seed": seed,+ "run_dir": run_dir,+ "elapsed_seconds": elapsed,+ "temperature": cfg.get("temperature"),+ "temperature_by_options": cfg.get("temperature_by_options", {}),+ "calibration": (cfg.get("training") or {}).get("laya_train_calibration"),+ "max_len": cfg.get("max_len"),+ "head_max_len": cfg.get("head_max_len"),+ }+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__)+ parser.add_argument("--gate", choices=sorted(GATES), required=True)+ parser.add_argument("--data", required=True)+ parser.add_argument("--out-dir", default="runs/step4")+ parser.add_argument("--base", default="convaiinnovations/laya")+ parser.add_argument("--device", default="auto")+ parser.add_argument("--losses", nargs="+", default=None,+ help="Override the gate's losses.")+ parser.add_argument("--seeds", nargs="+", type=int, default=None,+ help="Override the gate's seeds.")+ args = parser.parse_args()++ matrix = GATES[args.gate]+ if args.losses:+ matrix = [(loss, seed) for loss, seed in matrix if loss in args.losses]+ if args.seeds:+ matrix = [(loss, seed) for loss, seed in matrix if seed in args.seeds]+ if not matrix:+ parser.error("the loss/seed filters left the gate empty")++ os.makedirs(args.out_dir, exist_ok=True)+ runs = [run_one(args.data, args.out_dir, loss, seed, args.base, args.device,+ NOTEBOOK_HYPERPARAMS)+ for loss, seed in matrix]++ manifest = {+ "gate": args.gate,+ "base": args.base,+ "data": args.data,+ "hyperparameters": NOTEBOOK_HYPERPARAMS,+ "runs": runs,+ }+ manifest_path = os.path.join(args.out_dir, f"manifest_{args.gate}.json")+ with open(manifest_path, "w", encoding="utf-8") as f:+ json.dump(manifest, f, indent=2)+ print(f"\nwrote {manifest_path}")+ return 0+++if __name__ == "__main__":+ sys.exit(main())