Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

research(step4): reproduce typed-decisions through laya-train

Bruce-Yii2a16e8033169
diff --git a/research/scripts/step4_benchmark/README.md b/research/scripts/step4_benchmark/README.md
new file mode 100644
index 00000000..ebcc2f92
--- /dev/null
+++ b/research/scripts/step4_benchmark/README.md
@@ -0,0 +1,149 @@
+# Step 4 benchmark: reproduce typed-decisions through `laya-train`
+
+Harness for #887 Step 4. Two evidence surfaces, kept separate:
+
+1. **Typed-decisions reproduction** — the fine-tune is reproduced through the shipped
+ `laya-train` CLI and scored on the official test split: **accuracy, ECE, Brier**. This is the
+ surface that verifies acceptance criterion 1.
+2. **20-option order stability** — measured on **`massive_intent.en` at 20 options**, not on
+ typed-decisions (whose `choice` questions are 4-option). The original issue names that surface
+ explicitly; do not substitute a "20-option typed-decisions" run.
+
+The default loss is still `rlcd`. Whether `soft-ce` should replace it is decided by the Gate D
+matrix below, not by a single seed.
+
+## Files
+
+| File | Purpose |
+|---|---|
+| `prepare_data.py` | `LocalLLaMA/typed-decisions` → `train.jsonl` / `test.jsonl` / `test_meta.json` |
+| `run_benchmark.py` | Runs `laya-train` for a gate's loss × seed matrix; writes a per-run manifest |
+| `evaluate.py` | Accuracy / ECE / Brier on the test split, under shipped and raw temperatures |
+| `metrics.py` | The notebook's metric definitions (shared by `evaluate.py` and the smoke test) |
+| `order_flip.py` | Inference-only 20-option flip rate on `massive_intent.en` |
+| `smoke_test.py` | CPU, weight-free: metric plumbing + CLI wiring + tiny end-to-end |
+
+## Exact benchmark contract
+
+Pinned from the published notebook (`notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb`)
+and #887's thread. Confirm against the base checkpoint config at run time (read once, record it).
+
+| | value | source |
+|---|---|---|
+| dataset | `LocalLLaMA/typed-decisions`, config `all` | notebook |
+| train split | `train` — 1,200 cases / 6,000 decisions | notebook |
+| test split | `test` — 400 cases / 2,000 decisions | notebook |
+| base checkpoint | `convaiinnovations/laya` | notebook |
+| loss (reproduction) | `rlcd` (`laya-train` default) | #887 |
+| epochs | 4 | notebook |
+| micro-batch / grad-accum | 8 / 8 — effective batch 64 single-device (notebook: 8 × 2 GPUs × 4) | notebook, `TrainConfig` |
+| learning rates | encoder 2.5e-5, head 1e-4, AdamW cosine | notebook |
+| `max_len` / `head_max_len` | 1024 / 256 | notebook, `docs/finetune.md` |
+| training seed | 0 | #887 |
+| calibration slice | held out inside `laya-train`: `calib_frac` 0.1 capped at 400, seed 20260922 | `TrainConfig`, notebook |
+| mixed precision | fp16 autocast on CUDA; gradient checkpointing follows it | notebook, `TrainConfig` |
+
+### Metric definitions (notebook cell 14)
+
+- **accuracy** — top option vs gold label, over every decision (choice + noul + score).
+- **ECE** — `laya.common.ece_score` of the top probability against correctness, 15 equal-width
+ bins, over every decision.
+- **Brier** — mean over **choice and noul** decisions of `sum((p - gold)**2)`; the notebook does
+ not append score decisions to this mean, so a score change never moves Brier. Kept as-is so the
+ number compares with the reference.
+- **soft accuracy** — mean over **choice and noul** of `sum(p * gold)`.
+
+### Reference numbers to compare against
+
+From #887 (same split and seed, notebooks linked in-thread):
+
+| run | accuracy | ECE | Brier |
+|---|---|---|---|
+| base `laya` | 0.362 | 0.174 | 0.329 |
+| `loss="rlcd"` (notebook objective) | 0.7715 | 0.150 | 0.050 |
+| `loss="soft-ce"` | 0.790 | 0.157 | 0.047 |
+| published `laya-typed-decisions` | 0.766 | 0.213 | 0.061 |
+
+Accuracy is the cleanest training-reproduction signal; ECE and Brier must name the calibration
+path that produced them (next section).
+
+### Calibration is not byte-comparable across contracts
+
+`laya.train` fits with `laya.calibrate.fit_temperature_map` — per type, plus per option-count
+bucket above a floor — under the runtime clamp `[TEMP_MIN, TEMP_MAX] = [0.5, 5.0]`. The old
+notebook fitted one LBFGS temperature per type under `[0.1, 10]`. So `evaluate.py` reports ECE and
+Brier under **both** paths (`--temperatures shipped` uses the checkpoint's own fit; `--temperatures
+raw` uses 1.0 everywhere). An ECE difference explained by the calibration-contract change is not a
+training non-reproduction; say which path produced each number.
+
+## Gates
+
+### Gate R — reproduction
+
+One run: `rlcd`, seed 0. Passes when accuracy lands on the published checkpoint / the maintainer's
+target within single-seed noise, and ECE/Brier are reported with their calibration path. Nothing
+about the default loss is concluded from Gate R.
+
+```bash
+python research/scripts/step4_benchmark/prepare_data.py --out-dir data/typed_decisions
+python research/scripts/step4_benchmark/run_benchmark.py --gate R \
+ --data data/typed_decisions/train.jsonl --out-dir runs/step4 --device cuda
+python research/scripts/step4_benchmark/evaluate.py \
+ --checkpoint runs/step4/rlcd_seed0 \
+ --test-meta data/typed_decisions/test_meta.json --device cuda
+```
+
+### Gate D — default-loss decision
+
+`rlcd` vs `soft-ce` over more than one seed. The existing soft-ce win is single-seed, so it is not
+enough to recommend changing a shipped default. Minimal matrix (seeds 0/1/2, 6 runs):
+
+```bash
+python research/scripts/step4_benchmark/run_benchmark.py --gate D \
+ --data data/typed_decisions/train.jsonl --out-dir runs/step4 --device cuda
+```
+
+Summarise paired per-decision outcomes across seeds, not just the two means.
+
+### 20-option order stability
+
+`massive_intent.en`, 20 options (gold + 19 distractors, seed 13, matching
+`research/scripts/build_benchmark_nb.py`). GuilhermeFusari's seed-0/1 numbers from `laya.train`
+already cover this surface; reference them unless a reviewer wants the number reproduced on a
+specific checkpoint, in which case run inference-only:
+
+```bash
+python research/scripts/step4_benchmark/order_flip.py \
+ --checkpoint <checkpoint> --per-lang 300 --out flip.json
+```
+
+This does not train. Do not start a second MASSIVE training campaign by inertia.
+
+## Worked example / smoke
+
+`smoke_test.py` runs on CPU with a tiny local checkpoint and a tiny typed-decisions-shaped JSONL.
+It proves the metric definitions against hand-computed values, the CLI command, and the loop from
+JSONL through `laya-train` to `evaluate.py`. It downloads nothing and trains nothing large.
+
+```bash
+python research/scripts/step4_benchmark/smoke_test.py
+```
+
+## Runtime (keep the three apart)
+
+- **Reference (notebook, 2×T4):** ~4–6 minutes for the demo's 6,000 decisions.
+- **Guilherme measured (one T4):** ~48–54 minutes per run — but that is the **MASSIVE 20-option**
+ experiment, not typed-decisions. Do not carry it over.
+- **Our measured:** fill in from `manifest_*.json` (`runs[].elapsed_seconds`) after a real run.
+
+Single-device `laya-train` uses micro-batch 8 × grad-accum 8 = effective 64; the notebook used
+8 × 2 GPUs × 4 = effective 64. Same effective batch, different step count (single device does
+~4% more optimizer steps on the same data).
+
+## Constraints
+
+- Does not modify `laya/train.py`, `laya/train_cli.py` or `tests/test_train.py` (open PRs
+ #967/#968/#982).
+- Uses `laya-train` as a subprocess, so it does not depend on training internals.
+- Reuses `laya.common.ece_score` and the notebook's own metric definitions rather than adding a
+ second evaluation framework.
diff --git a/research/scripts/step4_benchmark/evaluate.py b/research/scripts/step4_benchmark/evaluate.py
new file mode 100644
index 00000000..a5fa9b67
--- /dev/null
+++ b/research/scripts/step4_benchmark/evaluate.py
@@ -0,0 +1,190 @@
+"""Evaluate a fine-tuned checkpoint on the typed-decisions test split.
+
+Mirrors cell 14 of the published notebook, restricted to the three metrics the Step-4 contract
+names: **accuracy**, **ECE** and **Brier**. The 20-option order-flip surface is deliberately not
+here: it belongs to `massive_intent.en`, not typed-decisions (see README).
+
+Calibration is not byte-comparable across contracts. `laya.train` fits temperatures with
+`fit_temperature_map` (per type, and per option-count bucket above a floor) under the runtime
+clamp `[0.5, 5.0]`; the old notebook fitted one LBFGS temperature per type under `[0.1, 10]`.
+So ECE and Brier are reported under **both** paths:
+
+- ``--temperatures shipped`` (default) — the checkpoint's own fitted `temperature` and
+ `temperature_by_options`, which is what the notebook's number also used;
+- ``--temperatures raw`` — temperature 1.0 everywhere, isolating the training from the
+ calibration contract.
+
+Accuracy is unaffected by temperature (it only rescales the logits), so it is the clean
+reproduction signal.
+
+Usage:
+ python evaluate.py --checkpoint runs/step4/rlcd-seed0 \
+ --test-meta data/typed_decisions/test_meta.json \
+ --out runs/step4/rlcd-seed0/eval_shipped.json
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from typing import Any, Dict, List
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from metrics import ( # noqa: E402
+ accuracy,
+ brier_score,
+ expected_calibration_error,
+ soft_accuracy,
+)
+
+
+def _normalise(values: List[float]) -> List[float]:
+ total = sum(values)
+ if total > 0:
+ return [v / total for v in values]
+ return values
+
+
+def evaluate_checkpoint(checkpoint_dir: str, test_meta: List[Dict[str, Any]],
+ device: str = "auto", temperatures: str = "shipped") -> Dict[str, Any]:
+ import laya
+
+ agent = laya.load(checkpoint_dir, device=device)
+ if temperatures == "raw":
+ agent.temperature = [1.0, 1.0, 1.0]
+ agent.temperature_by_options = {}
+ elif temperatures != "shipped":
+ raise ValueError("temperatures must be 'shipped' or 'raw', got %r" % (temperatures,))
+
+ corrects: List[float] = []
+ confidences: List[float] = []
+ briers: List[float] = []
+ softs: List[float] = []
+ per_type: Dict[str, List[float]] = {"choice": [], "noul": [], "score": []}
+ score_maes: List[float] = []
+ within_one: List[float] = []
+
+ for i, case in enumerate(test_meta):
+ state = case["state"]
+ questions = case["questions"]
+ gold = case["gold"]
+ answers = agent.predict(state, questions)["answers"]
+
+ for qid, qdef in questions.items():
+ if qid not in gold:
+ continue
+ g = gold[qid]
+ qtype = qdef["type"]
+ p_ans = answers[qid]
+
+ if qtype == "choice":
+ keys = list(qdef["criteria"].keys())
+ pred = p_ans["choice"]
+ gold_label = str(g["label"])
+ is_corr = float(pred == gold_label)
+
+ p_probs = _normalise([p_ans["probabilities"].get(k, 1e-6) for k in keys])
+ g_probs = _normalise([g["probabilities"].get(k, 1e-6) for k in keys])
+
+ corrects.append(is_corr)
+ confidences.append(max(p_probs))
+ briers.append(brier_score(p_probs, g_probs))
+ softs.append(soft_accuracy(p_probs, g_probs))
+ per_type["choice"].append(is_corr)
+
+ elif qtype == "noul":
+ p_val = float(p_ans["noul"])
+ g_val = float(g.get("noul", g.get("probabilities", {}).get("true", 0.5)))
+ gold_label = str(g["label"]).lower()
+ pred_label = "true" if p_val >= 0.5 else "false"
+ is_corr = float(pred_label == gold_label)
+
+ p_dist = [1.0 - p_val, p_val]
+ g_dist = [1.0 - g_val, g_val]
+
+ corrects.append(is_corr)
+ confidences.append(max(p_val, 1.0 - p_val))
+ briers.append(brier_score(p_dist, g_dist))
+ softs.append(soft_accuracy(p_dist, g_dist))
+ per_type["noul"].append(is_corr)
+
+ elif qtype == "score":
+ p_score = float(p_ans["score"])
+ g_score = float(g.get("score", 0.0))
+ score_maes.append(abs(p_score - g_score))
+ within_one.append(float(abs(p_score - g_score) <= 1.0))
+
+ n_levels = len(qdef.get("criteria", []))
+ p_probs = [p_ans["probabilities"].get(str(i), 0.0) for i in range(n_levels)]
+ if sum(p_probs) > 0:
+ p_probs = _normalise(p_probs)
+ p_lvl = int(max(range(len(p_probs)), key=lambda j: p_probs[j]))
+ confidences.append(max(p_probs))
+ else:
+ p_lvl = int(round(p_score))
+ confidences.append(0.5)
+
+ g_lvl = int(g.get("label", int(round(g_score))))
+ is_corr = float(p_lvl == g_lvl)
+ corrects.append(is_corr)
+ per_type["score"].append(is_corr)
+
+ if (i + 1) % 100 == 0:
+ print(f" {i + 1}/{len(test_meta)} cases", flush=True)
+
+ result = {
+ "checkpoint": checkpoint_dir,
+ "temperatures": temperatures,
+ "n_cases": len(test_meta),
+ "n_decisions": len(corrects),
+ "accuracy": round(accuracy(corrects), 4),
+ "ece": round(expected_calibration_error(confidences, corrects), 4),
+ "brier": round(float(sum(briers) / len(briers)) if briers else float("nan"), 4),
+ "soft_accuracy": round(float(sum(softs) / len(softs)) if softs else float("nan"), 4),
+ "score_mae": round(float(sum(score_maes) / len(score_maes)) if score_maes else float("nan"), 4),
+ "within_one_level": round(float(sum(within_one) / len(within_one)) if within_one else float("nan"), 4),
+ "per_type_accuracy": {k: round(accuracy(v), 4) for k, v in per_type.items() if v},
+ }
+ # `brier`/`soft_accuracy`/`ece` mix question types; record the counts they rest on.
+ result["n_brier_decisions"] = len(briers)
+ return result
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--checkpoint", required=True)
+ parser.add_argument("--test-meta", required=True)
+ parser.add_argument("--device", default="auto")
+ parser.add_argument("--temperatures", choices=["shipped", "raw"], default="shipped")
+ parser.add_argument("--out", default=None,
+ help="Output JSON (default: <checkpoint>/eval_<temperatures>.json)")
+ args = parser.parse_args()
+
+ with open(args.test_meta, encoding="utf-8") as f:
+ test_meta = json.load(f)
+ print(f"loaded {len(test_meta)} test cases", flush=True)
+
+ result = evaluate_checkpoint(args.checkpoint, test_meta, device=args.device,
+ temperatures=args.temperatures)
+
+ print("\n" + "=" * 60)
+ print(f"evaluation: {args.checkpoint} ({args.temperatures} temperatures)")
+ print("=" * 60)
+ print(f" accuracy: {result['accuracy']:.4f}")
+ print(f" ECE: {result['ece']:.4f}")
+ print(f" Brier: {result['brier']:.4f}")
+ print(f" soft acc: {result['soft_accuracy']:.4f}")
+ print(f" per type: {result['per_type_accuracy']}")
+ print("=" * 60)
+
+ out_path = args.out or os.path.join(args.checkpoint, f"eval_{args.temperatures}.json")
+ os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)
+ with open(out_path, "w", encoding="utf-8") as f:
+ json.dump(result, f, indent=2)
+ print(f"wrote {out_path}")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/research/scripts/step4_benchmark/metrics.py b/research/scripts/step4_benchmark/metrics.py
new file mode 100644
index 00000000..4871efc9
--- /dev/null
+++ b/research/scripts/step4_benchmark/metrics.py
@@ -0,0 +1,50 @@
+"""Metric definitions for the Step-4 benchmark, mirroring the published notebook.
+
+Reference: cell 14 of `notebooks/laya_finetune_typed_decisions_2xT4_kaggle.ipynb`. The published
+0.766 checkpoint and the maintainer's reference table are scored with these definitions, so a
+reproduction has to use them too, not a differently-defined library metric:
+
+- **accuracy** — top reported option against the gold label, over every decision (choice, score
+ and noul).
+- **ECE** — `laya.common.ece_score` of the top probability against correctness, 15 equal-width
+ bins, over every decision.
+- **brier** — mean over **choice and noul** decisions of `sum((p - gold)**2)`. The notebook does
+ not append score decisions to this mean, so a score change never moves Brier; kept as-is so the
+ number compares with the reference.
+- **soft_accuracy** — mean over **choice and noul** of `sum(p * gold)`.
+
+`gold` is the teacher distribution (`gold.probabilities`), not a one-hot label, so Brier and soft
+accuracy measure the distribution the run was trained to match.
+"""
+from __future__ import annotations
+
+from typing import Sequence
+
+import numpy as np
+
+from laya.common import ece_score
+
+
+def accuracy(correct: Sequence[float]) -> float:
+ return float(np.mean(correct)) if len(correct) else float("nan")
+
+
+def expected_calibration_error(confidences: Sequence[float], correct: Sequence[float],
+ bins: int = 15) -> float:
+ """Thin wrapper over the repository's `common.ece_score`, as the notebook calls it."""
+ if not len(confidences):
+ return float("nan")
+ return float(ece_score(np.asarray(confidences, dtype=float),
+ np.asarray(correct, dtype=float), bins=bins))
+
+
+def brier_score(probs: Sequence[float], gold: Sequence[float]) -> float:
+ p = np.asarray(probs, dtype=float)
+ g = np.asarray(gold, dtype=float)
+ return float(((p - g) ** 2).sum())
+
+
+def soft_accuracy(probs: Sequence[float], gold: Sequence[float]) -> float:
+ p = np.asarray(probs, dtype=float)
+ g = np.asarray(gold, dtype=float)
+ return float((p * g).sum())
diff --git a/research/scripts/step4_benchmark/prepare_data.py b/research/scripts/step4_benchmark/prepare_data.py
new file mode 100644
index 00000000..be9a6f30
--- /dev/null
+++ b/research/scripts/step4_benchmark/prepare_data.py
@@ -0,0 +1,96 @@
+"""Prepare LocalLLaMA/typed-decisions for `laya-train` and for evaluation.
+
+The benchmark's own rows are `{state, questions, gold}` with the teacher distribution per
+question. That is exactly the schema `laya.train` reads, so the conversion is a parse, not a
+reshape: the two JSON-string columns are decoded and the row is written back out as JSONL.
+
+Splits:
+
+* ``train.jsonl`` — the dataset's ``train`` split (1,200 cases / 6,000 decisions), the input to
+ ``laya-train``. The calibration slice is held out *inside* ``laya-train`` at a fixed seed
+ (``calib_frac`` 0.1, capped at 400, seed 20260922), matching the published notebook.
+* ``test.jsonl`` / ``test_meta.json`` — the dataset's ``test`` split (400 cases / 2,000
+ decisions), used by ``evaluate.py``. ``test_meta.json`` keeps ``id`` and ``workflow`` so
+ per-workflow accuracy can be reported alongside the pooled number.
+
+Usage:
+ python prepare_data.py --out-dir data/typed_decisions
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from typing import Any, Dict, List
+
+
+def _parse(value: Any) -> Any:
+ if isinstance(value, str):
+ return json.loads(value)
+ return value
+
+
+def case_to_record(row: Dict[str, Any]) -> Dict[str, Any]:
+ """One dataset row as a `laya-train` record (`{state, questions, gold}`)."""
+ return {
+ "state": _parse(row["state"]),
+ "questions": _parse(row["questions"]),
+ "gold": _parse(row["gold"]),
+ }
+
+
+def load_split(split: str, dataset: str = "LocalLLaMA/typed-decisions") -> List[Dict[str, Any]]:
+ try:
+ from datasets import load_dataset
+ except ImportError:
+ print("ERROR: the datasets package is required (pip install datasets)", file=sys.stderr)
+ sys.exit(1)
+ ds = load_dataset(dataset, "all", split=split)
+ return list(ds)
+
+
+def write_split(rows: List[Dict[str, Any]], out_dir: str, split: str) -> int:
+ records = [case_to_record(r) for r in rows]
+ path = os.path.join(out_dir, f"{split}.jsonl")
+ with open(path, "w", encoding="utf-8") as f:
+ for rec in records:
+ f.write(json.dumps(rec, ensure_ascii=False) + "\n")
+ return len(records)
+
+
+def write_test_meta(rows: List[Dict[str, Any]], out_dir: str) -> int:
+ meta = []
+ for r in rows:
+ meta.append({
+ "id": r.get("id", ""),
+ "workflow": r.get("workflow", ""),
+ "state": _parse(r["state"]),
+ "questions": _parse(r["questions"]),
+ "gold": _parse(r["gold"]),
+ })
+ path = os.path.join(out_dir, "test_meta.json")
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump(meta, f, indent=2)
+ return len(meta)
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--out-dir", default="data/typed_decisions")
+ parser.add_argument("--dataset", default="LocalLLaMA/typed-decisions")
+ args = parser.parse_args()
+
+ os.makedirs(args.out_dir, exist_ok=True)
+ for split in ("train", "test"):
+ rows = load_split(split, args.dataset)
+ n = write_split(rows, args.out_dir, split)
+ print(f"{split}: {n} cases -> {os.path.join(args.out_dir, split + '.jsonl')}")
+ if split == "test":
+ m = write_test_meta(rows, args.out_dir)
+ print(f"test meta: {m} cases -> {os.path.join(args.out_dir, 'test_meta.json')}")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/research/scripts/step4_benchmark/run_benchmark.py b/research/scripts/step4_benchmark/run_benchmark.py
new file mode 100644
index 00000000..ca411700
--- /dev/null
+++ b/research/scripts/step4_benchmark/run_benchmark.py
@@ -0,0 +1,150 @@
+"""Run the Step-4 benchmark through the `laya-train` CLI.
+
+Two gates, kept separate on purpose:
+
+* **Gate R — reproduction.** Seed 0, the notebook's hyperparameters, `--loss rlcd`. It has to land
+ on the published checkpoint / the maintainer's target within noise; that verifies acceptance
+ criterion 1. Nothing else is claimed from one seed.
+* **Gate D — default-loss decision.** `rlcd` vs `soft-ce` over more than one seed. The existing
+ single-seed soft-ce win is not enough to recommend changing a shipped default, so Gate D is
+ only designed and run after Gate R passes.
+
+This calls the CLI as a subprocess: it does not import `laya.train`, so it cannot depend on the
+training internals #967/#968/#982 are still changing.
+
+Usage:
+ # Gate R: one run
+ python run_benchmark.py --gate R --data data/typed_decisions/train.jsonl \
+ --out-dir runs/step4 --device cuda
+
+ # Gate D: rlcd + soft-ce over seeds 0/1/2
+ python run_benchmark.py --gate D --data data/typed_decisions/train.jsonl \
+ --out-dir runs/step4 --device cuda
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import shlex
+import subprocess
+import sys
+import time
+from typing import Any, Dict, List, Tuple
+
+# The notebook's recipe, single-device form. 8 micro-batch x 2 GPUs x 4 accumulation = 64
+# effective; on one device that is 8 x 8.
+NOTEBOOK_HYPERPARAMS = {
+ "epochs": 4,
+ "micro_batch": 8,
+ "grad_accum": 8,
+ "encoder_lr": 2.5e-5,
+ "head_lr": 1e-4,
+ "max_len": 1024,
+ "head_max_len": 256,
+}
+
+GATES: Dict[str, List[Tuple[str, int]]] = {
+ "R": [("rlcd", 0)],
+ "D": [("rlcd", 0), ("rlcd", 1), ("rlcd", 2),
+ ("soft-ce", 0), ("soft-ce", 1), ("soft-ce", 2)],
+}
+
+
+def build_command(data: str, out_dir: str, loss: str, seed: int, base: str, device: str,
+ hyper: Dict[str, Any]) -> List[str]:
+ return [
+ sys.executable, "-m", "laya.train_cli",
+ "--data", data,
+ "--out", out_dir,
+ "--base", base,
+ "--loss", loss,
+ "--seed", str(seed),
+ "--epochs", str(hyper["epochs"]),
+ "--micro-batch", str(hyper["micro_batch"]),
+ "--grad-accum", str(hyper["grad_accum"]),
+ "--encoder-lr", str(hyper["encoder_lr"]),
+ "--head-lr", str(hyper["head_lr"]),
+ "--max-len", str(hyper["max_len"]),
+ "--head-max-len", str(hyper["head_max_len"]),
+ "--device", device,
+ ]
+
+
+def run_one(data: str, out_root: str, loss: str, seed: int, base: str, device: str,
+ hyper: Dict[str, Any]) -> Dict[str, Any]:
+ run_dir = os.path.join(out_root, f"{loss.replace('-', '_')}-seed{seed}")
+ os.makedirs(run_dir, exist_ok=True)
+ cmd = build_command(data, run_dir, loss, seed, base, device, hyper)
+ print("\n" + "=" * 72)
+ print(" ".join(shlex.quote(c) for c in cmd))
+ print("=" * 72, flush=True)
+
+ t0 = time.time()
+ result = subprocess.run(cmd, text=True)
+ elapsed = round(time.time() - t0, 1)
+ if result.returncode != 0:
+ raise SystemExit(f"laya-train failed for loss={loss} seed={seed} "
+ f"(exit {result.returncode})")
+
+ cfg_path = os.path.join(run_dir, "rl_agent_config.json")
+ cfg: Dict[str, Any] = {}
+ if os.path.exists(cfg_path):
+ with open(cfg_path, encoding="utf-8") as f:
+ cfg = json.load(f)
+
+ return {
+ "loss": loss,
+ "seed": seed,
+ "run_dir": run_dir,
+ "elapsed_seconds": elapsed,
+ "temperature": cfg.get("temperature"),
+ "temperature_by_options": cfg.get("temperature_by_options", {}),
+ "calibration": (cfg.get("training") or {}).get("laya_train_calibration"),
+ "max_len": cfg.get("max_len"),
+ "head_max_len": cfg.get("head_max_len"),
+ }
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__)
+ parser.add_argument("--gate", choices=sorted(GATES), required=True)
+ parser.add_argument("--data", required=True)
+ parser.add_argument("--out-dir", default="runs/step4")
+ parser.add_argument("--base", default="convaiinnovations/laya")
+ parser.add_argument("--device", default="auto")
+ parser.add_argument("--losses", nargs="+", default=None,
+ help="Override the gate's losses.")
+ parser.add_argument("--seeds", nargs="+", type=int, default=None,
+ help="Override the gate's seeds.")
+ args = parser.parse_args()
+
+ matrix = GATES[args.gate]
+ if args.losses:
+ matrix = [(loss, seed) for loss, seed in matrix if loss in args.losses]
+ if args.seeds:
+ matrix = [(loss, seed) for loss, seed in matrix if seed in args.seeds]
+ if not matrix:
+ parser.error("the loss/seed filters left the gate empty")
+
+ os.makedirs(args.out_dir, exist_ok=True)
+ runs = [run_one(args.data, args.out_dir, loss, seed, args.base, args.device,
+ NOTEBOOK_HYPERPARAMS)
+ for loss, seed in matrix]
+
+ manifest = {
+ "gate": args.gate,
+ "base": args.base,
+ "data": args.data,
+ "hyperparameters": NOTEBOOK_HYPERPARAMS,
+ "runs": runs,
+ }
+ manifest_path = os.path.join(args.out_dir, f"manifest_{args.gate}.json")
+ with open(manifest_path, "w", encoding="utf-8") as f:
+ json.dump(manifest, f, indent=2)
+ print(f"\nwrote {manifest_path}")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())