Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.
Python calibrationclassificationdecision-modelhuggingfacejevmodernbertmultilingualnlppythonpytorchroutingtyped-decisionszero-shot
research(step4): three-seed RLCD vs soft-CE panel and verdict
Bruce-Yii331576c3772e
diff --git a/research/scripts/step4_benchmark/GATE_D_RESULTS.md b/research/scripts/step4_benchmark/GATE_D_RESULTS.mdnew file mode 100644index 00000000..44521ece--- /dev/null+++ b/research/scripts/step4_benchmark/GATE_D_RESULTS.md@@ -0,0 +1,128 @@+# Gate D — RLCD vs soft-CE default-loss panel (three seeds)++Evidence for [#887](https://github.com/NandhaKishorM/laya/issues/887) Step 4. The matrix exists to+answer one question: does the single-seed soft-CE win hold up well enough to change the shipped+default loss? It does not.++- Upstream pin: `a4a8921afebfd852bba0000475cfb6ab737a124c`+- Compute: one Tesla T4 per run (`CUDA_VISIBLE_DEVICES=0`, exactly one visible GPU asserted),+ `torch 2.11.0+cu128`, Kaggle free tier.+- Contract: the Gate R contract — `LocalLLaMA/typed-decisions` config `all`, 1,200 train cases /+ 6,000 decisions, calibration 400 items (`calib_frac` 0.1, `calib_seed` 20260922, held out inside+ `laya-train`), 400 test cases / 2,000 decisions, 4 epochs, micro-batch 8 × grad-accum 8,+ encoder LR 2.5e-5 / head LR 1e-4, `max_len` 1024 / `head_max_len` 256, no shuffling, no early+ stopping, `loss` the only varying field.+- `rlcd` seed 0 is the Gate R run, reused here; the other five cells were trained for this panel.+- Raw per-cell artifacts: [`results/step4_gate_d_results.json`](results/step4_gate_d_results.json),+ [`results/step4_gate_r_results.json`](results/step4_gate_r_results.json).++## Result++| loss | seed | accuracy | ECE | Brier | wall (s) |+|---|---|---|---|---|---|+| `rlcd` | 0 | 0.7750 | 0.1504 | 0.0519 | 2691.3 |+| `rlcd` | 1 | 0.7780 | 0.1477 | 0.0519 | 2697.6 |+| `rlcd` | 2 | 0.7755 | 0.1530 | 0.0490 | 2708.5 |+| `soft-ce` | 0 | 0.7605 | 0.1353 | 0.0532 | 2691.4 |+| `soft-ce` | 1 | 0.7915 | 0.1574 | 0.0481 | 2705.1 |+| `soft-ce` | 2 | 0.7770 | 0.1471 | 0.0513 | 2701.8 |++Same split and seed, from the #887 thread, for context:++| run | accuracy | ECE | Brier |+|---|---|---|---|+| base `convaiinnovations/laya` | 0.3620 | 0.1741 | 0.3287 |+| published `laya-typed-decisions` | 0.766 | 0.213 | 0.061 |+| `rlcd` single seed (#887) | 0.7715 | 0.150 | 0.050 |+| `soft-ce` single seed (#887) | 0.790 | 0.157 | 0.047 |++## Paired `soft-ce − rlcd`++| seed | accuracy Δ | ECE Δ | Brier Δ |+|---|---|---|---|+| 0 | −0.0145 | −0.0151 | +0.0013 |+| 1 | +0.0135 | +0.0097 | −0.0038 |+| 2 | +0.0015 | −0.0059 | +0.0023 |+| mean | **+0.0002** | −0.0038 | −0.0001 |+| min / max | −0.0145 / +0.0135 | −0.0151 / +0.0097 | −0.0038 / +0.0023 |++Across the three seeds `rlcd` accuracy is 0.7762 (std 0.0016) and `soft-ce` is 0.7763+(std 0.0155). The paired delta changes sign between seeds, so there is no consistent direction.++ECE and Brier are reported under the shipped temperature fit (see limitations) and only over the+notebook's Brier scope (choice + noul), so small movements there are not portable.++## Per-type accuracy (mean over seeds)++| type | `rlcd` | `soft-ce` |+|---|---|---|+| choice | 0.7422 | 0.7539 |+| noul | 0.8478 | 0.8533 |+| score | 0.7479 | 0.7354 |++The small choice/noul gain and the score loss swap rank between seeds; neither is stable.++## Verdict++`INSUFFICIENT`. The paired accuracy delta changes sign across seeds and its mean (+0.0002) is far+below the 0.005 bar the analyser treats as material, while the soft-CE seed spread (std 0.0155) is+an order of magnitude larger than RLCD's (0.0016). Three training seeds cannot support changing a+shipped default in either direction, so `rlcd` remains the conservative default. This PR does not+change the default.++## Reproduction++The harness runs `laya-train` as a subprocess and scores the test split with the notebook's metric+definitions. Locally (no GPU, weight-free):++```bash+python research/scripts/step4_benchmark/smoke_test.py+python research/scripts/step4_benchmark/analyze_gate_d.py \+ --results research/scripts/step4_benchmark/results/step4_gate_d_results.json+```++The six cells were trained on Kaggle from the committed generators, one process per GPU:++```bash+python research/scripts/step4_benchmark/kaggle/build_kernel.py # Gate R (reused rlcd seed 0)+python research/scripts/step4_benchmark/kaggle/build_gate_d.py # Gate D (remaining 5 cells)+kaggle kernels push -p research/scripts/step4_benchmark/kaggle+kaggle kernels output yuyi722333/laya-step-4-gate-d -p out --file-pattern 'step4_gate_d_results\.json'+```++Each cell asserts the fail-closed readback (device count, `max_len`/`head_max_len` 1024/256,+dataset counts, finite metrics) before its numbers are accepted.++## Limitations++- **Notebook RNG mismatch.** `laya.train` derives the epoch seed as `seed + epoch`; the published+ notebook used `42 + epoch + rank`. The gap to the notebook's own numbers is therefore not a pure+ environment delta.+- **Environment.** `torch 2.11.0+cu128` here; `docs/finetune.md` describes 2.14.+- **Calibration scope.** ECE uses the checkpoint's shipped per-type temperature fit under the+ runtime clamp `[0.5, 5.0]`; the notebook fitted per-type LBFGS temperatures under `[0.1, 10]`.+ The two are not byte-comparable, so ECE/Brier name their path. Brier follows the notebook and+ averages choice + noul only, so a score change never moves it.+- **Base config metadata.** The base checkpoint's `training` block records `max_len` 512 /+ `head_max_len` 192; each run passed the contract's explicit 1024 / 256 and asserted it back.+- **Inherited metadata.** `training.*` counters (`updates`, `epochs_completed`, `hours`) are copied+ from the base checkpoint into the fine-tuned checkpoint.+- **Peak VRAM** was not captured.+- **n = 3 training seeds.** No significance claim is made from three seeds.++## Reused evidence, not rerun++20-option order stability is a different surface (typed-decisions `choice` questions have four+options). This PR does not train MASSIVE. It references+[GuilhermeFusari's public notebook *Laya: option shuffling at 20 options (MASSIVE intent,+English)*](https://www.kaggle.com/code/guilhermediasfusari/laya-option-shuffling-at-20-options-massive-inte)+as the order-stability evidence: `massive_intent.en`, gold intent plus 19 distractors drawn from+the 60 intents in random order, seed 13, `max_len` 512, one process per T4, on Laya+`8a6e132`. Our numbers here are on current main (`a4a8921`), so the two are reported separately+rather than merged.++## Follow-up candidates (not in this PR)++- The `training.*` inheritance above is a checkpoint-metadata defect, not a training defect; it+ would need its own issue.+- The notebook-vs-`laya.train` epoch-seed formula is a reproducibility mismatch, also separate.diff --git a/research/scripts/step4_benchmark/README.md b/research/scripts/step4_benchmark/README.mdindex 27fc8000..9287e697 100644--- a/research/scripts/step4_benchmark/README.md+++ b/research/scripts/step4_benchmark/README.md@@ -105,7 +105,9 @@ python research/scripts/step4_benchmark/run_benchmark.py --gate D \--data data/typed_decisions/train.jsonl --out-dir runs/step4 --device cuda```-Summarise paired per-decision outcomes across seeds, not just the two means.+Summarise paired per-decision outcomes across seeds, not just the two means. The measured+three-seed panel and its verdict are in [`GATE_D_RESULTS.md`](GATE_D_RESULTS.md), with the raw cells+under [`results/`](results/).### 20-option order stabilitydiff --git a/research/scripts/step4_benchmark/results/step4_gate_d_results.json b/research/scripts/step4_benchmark/results/step4_gate_d_results.jsonnew file mode 100644index 00000000..3eb8967e--- /dev/null+++ b/research/scripts/step4_benchmark/results/step4_gate_d_results.json@@ -0,0 +1,308 @@+{+ "meta": {+ "pinned_sha": "a4a8921afebfd852bba0000475cfb6ab737a124c",+ "torch": "2.11.0+cu128",+ "gpu": "Tesla T4",+ "visible_gpus": 1,+ "base_config": {+ "max_len": 512,+ "head_max_len": 192+ },+ "n_train_cases": 1200,+ "n_test_cases": 400,+ "reused": {+ "rlcd_seed0": "Gate R kernel (separate session, same SHA/data/command)"+ }+ },+ "rlcd_seed0": {+ "checkpoint": "Gate R kernel run_rlcd_seed0 (reused)",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.775,+ "ece": 0.1504,+ "brier": 0.0519,+ "soft_accuracy": 0.5654,+ "per_type_accuracy": {+ "choice": 0.7417,+ "noul": 0.8517,+ "score": 0.7425+ },+ "loss": "rlcd",+ "seed": 0,+ "wall_clock_s": 2691.3+ },+ "base": {+ "checkpoint": "convaiinnovations/laya",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.362,+ "ece": 0.1741,+ "brier": 0.3287,+ "soft_accuracy": 0.3656,+ "score_mae": 0.6936,+ "within_one_level": 0.7525,+ "per_type_accuracy": {+ "choice": 0.29,+ "noul": 0.4867,+ "score": 0.3225+ },+ "n_brier_decisions": 1200+ },+ "rlcd_seed1": {+ "checkpoint": "/kaggle/working/run_rlcd_seed1",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.778,+ "ece": 0.1477,+ "brier": 0.0519,+ "soft_accuracy": 0.5652,+ "score_mae": 0.2155,+ "within_one_level": 0.9888,+ "per_type_accuracy": {+ "choice": 0.74,+ "noul": 0.8467,+ "score": 0.755+ },+ "n_brier_decisions": 1200,+ "loss": "rlcd",+ "seed": 1,+ "wall_clock_s": 2697.6,+ "temperature": [+ 1.0706707239151,+ 1.0655159950256348,+ 1.1516488790512085+ ],+ "calibration": {+ "choice": {+ "items": 141,+ "temperature": 1.0706707239151,+ "issues": [],+ "accuracy": 0.7588652482269503,+ "mean_confidence": 0.6076728429612874+ },+ "score": {+ "items": 153,+ "temperature": 1.0655159950256348,+ "issues": [],+ "accuracy": 0.7843137254901961,+ "mean_confidence": 0.5692810567559323+ },+ "noul": {+ "items": 106,+ "temperature": 1.1516488790512085,+ "issues": [],+ "accuracy": 0.8584905660377359,+ "mean_confidence": 0.7542289320023828+ }+ },+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",+ "finite": true+ },+ "soft-ce_seed0": {+ "checkpoint": "/kaggle/working/run_soft_ce_seed0",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.7605,+ "ece": 0.1353,+ "brier": 0.0532,+ "soft_accuracy": 0.5642,+ "score_mae": 0.2456,+ "within_one_level": 0.9938,+ "per_type_accuracy": {+ "choice": 0.7417,+ "noul": 0.855,+ "score": 0.7037+ },+ "n_brier_decisions": 1200,+ "loss": "soft-ce",+ "seed": 0,+ "wall_clock_s": 2691.4,+ "temperature": [+ 1.0631763935089111,+ 1.0451915264129639,+ 1.0851869583129883+ ],+ "calibration": {+ "choice": {+ "items": 141,+ "temperature": 1.0631763935089111,+ "issues": [],+ "accuracy": 0.7943262411347518,+ "mean_confidence": 0.6069391803631538+ },+ "score": {+ "items": 153,+ "temperature": 1.0451915264129639,+ "issues": [],+ "accuracy": 0.7843137254901961,+ "mean_confidence": 0.5572438980445797+ },+ "noul": {+ "items": 106,+ "temperature": 1.0851869583129883,+ "issues": [],+ "accuracy": 0.839622641509434,+ "mean_confidence": 0.7526641343832678+ }+ },+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",+ "finite": true+ },+ "soft-ce_seed1": {+ "checkpoint": "/kaggle/working/run_soft_ce_seed1",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.7915,+ "ece": 0.1574,+ "brier": 0.0481,+ "soft_accuracy": 0.5705,+ "score_mae": 0.2105,+ "within_one_level": 0.9912,+ "per_type_accuracy": {+ "choice": 0.77,+ "noul": 0.8583,+ "score": 0.7575+ },+ "n_brier_decisions": 1200,+ "loss": "soft-ce",+ "seed": 1,+ "wall_clock_s": 2705.1,+ "temperature": [+ 1.0398290157318115,+ 1.0743035078048706,+ 1.139175534248352+ ],+ "calibration": {+ "choice": {+ "items": 141,+ "temperature": 1.0398290157318115,+ "issues": [],+ "accuracy": 0.8014184397163121,+ "mean_confidence": 0.615891238130113+ },+ "score": {+ "items": 153,+ "temperature": 1.0743035078048706,+ "issues": [],+ "accuracy": 0.7777777777777778,+ "mean_confidence": 0.5753281628764587+ },+ "noul": {+ "items": 106,+ "temperature": 1.139175534248352,+ "issues": [],+ "accuracy": 0.839622641509434,+ "mean_confidence": 0.7543471859560601+ }+ },+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",+ "finite": true+ },+ "rlcd_seed2": {+ "checkpoint": "/kaggle/working/run_rlcd_seed2",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.7755,+ "ece": 0.153,+ "brier": 0.049,+ "soft_accuracy": 0.5638,+ "score_mae": 0.2195,+ "within_one_level": 0.9938,+ "per_type_accuracy": {+ "choice": 0.745,+ "noul": 0.845,+ "score": 0.7462+ },+ "n_brier_decisions": 1200,+ "loss": "rlcd",+ "seed": 2,+ "wall_clock_s": 2708.5,+ "temperature": [+ 1.112676978111267,+ 1.054789662361145,+ 1.1128166913986206+ ],+ "calibration": {+ "choice": {+ "items": 141,+ "temperature": 1.112676978111267,+ "issues": [],+ "accuracy": 0.7730496453900709,+ "mean_confidence": 0.5887910391183622+ },+ "score": {+ "items": 153,+ "temperature": 1.054789662361145,+ "issues": [],+ "accuracy": 0.7908496732026143,+ "mean_confidence": 0.5640255503811176+ },+ "noul": {+ "items": 106,+ "temperature": 1.1128166913986206,+ "issues": [],+ "accuracy": 0.8584905660377359,+ "mean_confidence": 0.7526793880094345+ }+ },+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",+ "finite": true+ },+ "soft-ce_seed2": {+ "checkpoint": "/kaggle/working/run_soft_ce_seed2",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.777,+ "ece": 0.1471,+ "brier": 0.0513,+ "soft_accuracy": 0.5667,+ "score_mae": 0.214,+ "within_one_level": 0.99,+ "per_type_accuracy": {+ "choice": 0.75,+ "noul": 0.8467,+ "score": 0.745+ },+ "n_brier_decisions": 1200,+ "loss": "soft-ce",+ "seed": 2,+ "wall_clock_s": 2701.8,+ "temperature": [+ 1.0387086868286133,+ 1.053942084312439,+ 1.08060884475708+ ],+ "calibration": {+ "choice": {+ "items": 141,+ "temperature": 1.0387086868286133,+ "issues": [],+ "accuracy": 0.7730496453900709,+ "mean_confidence": 0.6111094674737355+ },+ "score": {+ "items": 153,+ "temperature": 1.053942084312439,+ "issues": [],+ "accuracy": 0.8300653594771242,+ "mean_confidence": 0.5717240346959584+ },+ "noul": {+ "items": 106,+ "temperature": 1.08060884475708,+ "issues": [],+ "accuracy": 0.8584905660377359,+ "mean_confidence": 0.7583364282950572+ }+ },+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",+ "finite": true+ }+}\ No newline at end of filediff --git a/research/scripts/step4_benchmark/results/step4_gate_r_results.json b/research/scripts/step4_benchmark/results/step4_gate_r_results.jsonnew file mode 100644index 00000000..96c55a19--- /dev/null+++ b/research/scripts/step4_benchmark/results/step4_gate_r_results.json@@ -0,0 +1,71 @@+{+ "meta": {+ "pinned_sha": "a4a8921afebfd852bba0000475cfb6ab737a124c",+ "torch": "2.11.0+cu128",+ "gpu": "Tesla T4",+ "visible_gpus": 1,+ "train_seconds": 2691.3,+ "peak_gpu_gb": 0.0,+ "train_split_line": "train items 5600, calibration items 400, skipped {}, device cuda",+ "base_config": {+ "max_len": 512,+ "head_max_len": 192+ },+ "n_train_cases": 1200,+ "n_test_cases": 400+ },+ "rlcd_seed0": {+ "checkpoint": "/kaggle/working/run_rlcd_seed0",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.775,+ "ece": 0.1504,+ "brier": 0.0519,+ "soft_accuracy": 0.5654,+ "score_mae": 0.2246,+ "within_one_level": 0.99,+ "per_type_accuracy": {+ "choice": 0.7417,+ "noul": 0.8517,+ "score": 0.7425+ },+ "n_brier_decisions": 1200+ },+ "base": {+ "checkpoint": "convaiinnovations/laya",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.362,+ "ece": 0.1741,+ "brier": 0.3287,+ "soft_accuracy": 0.3656,+ "score_mae": 0.6936,+ "within_one_level": 0.7525,+ "per_type_accuracy": {+ "choice": 0.29,+ "noul": 0.4867,+ "score": 0.3225+ },+ "n_brier_decisions": 1200+ },+ "published": {+ "checkpoint": "convaiinnovations/laya-typed-decisions",+ "temperatures": "shipped",+ "n_cases": 400,+ "n_decisions": 2000,+ "accuracy": 0.7665,+ "ece": 0.2137,+ "brier": 0.0658,+ "soft_accuracy": 0.5087,+ "score_mae": 0.2424,+ "within_one_level": 0.995,+ "per_type_accuracy": {+ "choice": 0.7333,+ "noul": 0.8567,+ "score": 0.7238+ },+ "n_brier_decisions": 1200+ }+}\ No newline at end of file