Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

research(step4): three-seed RLCD vs soft-CE panel and verdict

Bruce-Yii331576c3772e
diff --git a/research/scripts/step4_benchmark/GATE_D_RESULTS.md b/research/scripts/step4_benchmark/GATE_D_RESULTS.md
new file mode 100644
index 00000000..44521ece
--- /dev/null
+++ b/research/scripts/step4_benchmark/GATE_D_RESULTS.md
@@ -0,0 +1,128 @@
+# Gate D — RLCD vs soft-CE default-loss panel (three seeds)
+
+Evidence for [#887](https://github.com/NandhaKishorM/laya/issues/887) Step 4. The matrix exists to
+answer one question: does the single-seed soft-CE win hold up well enough to change the shipped
+default loss? It does not.
+
+- Upstream pin: `a4a8921afebfd852bba0000475cfb6ab737a124c`
+- Compute: one Tesla T4 per run (`CUDA_VISIBLE_DEVICES=0`, exactly one visible GPU asserted),
+ `torch 2.11.0+cu128`, Kaggle free tier.
+- Contract: the Gate R contract — `LocalLLaMA/typed-decisions` config `all`, 1,200 train cases /
+ 6,000 decisions, calibration 400 items (`calib_frac` 0.1, `calib_seed` 20260922, held out inside
+ `laya-train`), 400 test cases / 2,000 decisions, 4 epochs, micro-batch 8 × grad-accum 8,
+ encoder LR 2.5e-5 / head LR 1e-4, `max_len` 1024 / `head_max_len` 256, no shuffling, no early
+ stopping, `loss` the only varying field.
+- `rlcd` seed 0 is the Gate R run, reused here; the other five cells were trained for this panel.
+- Raw per-cell artifacts: [`results/step4_gate_d_results.json`](results/step4_gate_d_results.json),
+ [`results/step4_gate_r_results.json`](results/step4_gate_r_results.json).
+
+## Result
+
+| loss | seed | accuracy | ECE | Brier | wall (s) |
+|---|---|---|---|---|---|
+| `rlcd` | 0 | 0.7750 | 0.1504 | 0.0519 | 2691.3 |
+| `rlcd` | 1 | 0.7780 | 0.1477 | 0.0519 | 2697.6 |
+| `rlcd` | 2 | 0.7755 | 0.1530 | 0.0490 | 2708.5 |
+| `soft-ce` | 0 | 0.7605 | 0.1353 | 0.0532 | 2691.4 |
+| `soft-ce` | 1 | 0.7915 | 0.1574 | 0.0481 | 2705.1 |
+| `soft-ce` | 2 | 0.7770 | 0.1471 | 0.0513 | 2701.8 |
+
+Same split and seed, from the #887 thread, for context:
+
+| run | accuracy | ECE | Brier |
+|---|---|---|---|
+| base `convaiinnovations/laya` | 0.3620 | 0.1741 | 0.3287 |
+| published `laya-typed-decisions` | 0.766 | 0.213 | 0.061 |
+| `rlcd` single seed (#887) | 0.7715 | 0.150 | 0.050 |
+| `soft-ce` single seed (#887) | 0.790 | 0.157 | 0.047 |
+
+## Paired `soft-ce − rlcd`
+
+| seed | accuracy Δ | ECE Δ | Brier Δ |
+|---|---|---|---|
+| 0 | −0.0145 | −0.0151 | +0.0013 |
+| 1 | +0.0135 | +0.0097 | −0.0038 |
+| 2 | +0.0015 | −0.0059 | +0.0023 |
+| mean | **+0.0002** | −0.0038 | −0.0001 |
+| min / max | −0.0145 / +0.0135 | −0.0151 / +0.0097 | −0.0038 / +0.0023 |
+
+Across the three seeds `rlcd` accuracy is 0.7762 (std 0.0016) and `soft-ce` is 0.7763
+(std 0.0155). The paired delta changes sign between seeds, so there is no consistent direction.
+
+ECE and Brier are reported under the shipped temperature fit (see limitations) and only over the
+notebook's Brier scope (choice + noul), so small movements there are not portable.
+
+## Per-type accuracy (mean over seeds)
+
+| type | `rlcd` | `soft-ce` |
+|---|---|---|
+| choice | 0.7422 | 0.7539 |
+| noul | 0.8478 | 0.8533 |
+| score | 0.7479 | 0.7354 |
+
+The small choice/noul gain and the score loss swap rank between seeds; neither is stable.
+
+## Verdict
+
+`INSUFFICIENT`. The paired accuracy delta changes sign across seeds and its mean (+0.0002) is far
+below the 0.005 bar the analyser treats as material, while the soft-CE seed spread (std 0.0155) is
+an order of magnitude larger than RLCD's (0.0016). Three training seeds cannot support changing a
+shipped default in either direction, so `rlcd` remains the conservative default. This PR does not
+change the default.
+
+## Reproduction
+
+The harness runs `laya-train` as a subprocess and scores the test split with the notebook's metric
+definitions. Locally (no GPU, weight-free):
+
+```bash
+python research/scripts/step4_benchmark/smoke_test.py
+python research/scripts/step4_benchmark/analyze_gate_d.py \
+ --results research/scripts/step4_benchmark/results/step4_gate_d_results.json
+```
+
+The six cells were trained on Kaggle from the committed generators, one process per GPU:
+
+```bash
+python research/scripts/step4_benchmark/kaggle/build_kernel.py # Gate R (reused rlcd seed 0)
+python research/scripts/step4_benchmark/kaggle/build_gate_d.py # Gate D (remaining 5 cells)
+kaggle kernels push -p research/scripts/step4_benchmark/kaggle
+kaggle kernels output yuyi722333/laya-step-4-gate-d -p out --file-pattern 'step4_gate_d_results\.json'
+```
+
+Each cell asserts the fail-closed readback (device count, `max_len`/`head_max_len` 1024/256,
+dataset counts, finite metrics) before its numbers are accepted.
+
+## Limitations
+
+- **Notebook RNG mismatch.** `laya.train` derives the epoch seed as `seed + epoch`; the published
+ notebook used `42 + epoch + rank`. The gap to the notebook's own numbers is therefore not a pure
+ environment delta.
+- **Environment.** `torch 2.11.0+cu128` here; `docs/finetune.md` describes 2.14.
+- **Calibration scope.** ECE uses the checkpoint's shipped per-type temperature fit under the
+ runtime clamp `[0.5, 5.0]`; the notebook fitted per-type LBFGS temperatures under `[0.1, 10]`.
+ The two are not byte-comparable, so ECE/Brier name their path. Brier follows the notebook and
+ averages choice + noul only, so a score change never moves it.
+- **Base config metadata.** The base checkpoint's `training` block records `max_len` 512 /
+ `head_max_len` 192; each run passed the contract's explicit 1024 / 256 and asserted it back.
+- **Inherited metadata.** `training.*` counters (`updates`, `epochs_completed`, `hours`) are copied
+ from the base checkpoint into the fine-tuned checkpoint.
+- **Peak VRAM** was not captured.
+- **n = 3 training seeds.** No significance claim is made from three seeds.
+
+## Reused evidence, not rerun
+
+20-option order stability is a different surface (typed-decisions `choice` questions have four
+options). This PR does not train MASSIVE. It references
+[GuilhermeFusari's public notebook *Laya: option shuffling at 20 options (MASSIVE intent,
+English)*](https://www.kaggle.com/code/guilhermediasfusari/laya-option-shuffling-at-20-options-massive-inte)
+as the order-stability evidence: `massive_intent.en`, gold intent plus 19 distractors drawn from
+the 60 intents in random order, seed 13, `max_len` 512, one process per T4, on Laya
+`8a6e132`. Our numbers here are on current main (`a4a8921`), so the two are reported separately
+rather than merged.
+
+## Follow-up candidates (not in this PR)
+
+- The `training.*` inheritance above is a checkpoint-metadata defect, not a training defect; it
+ would need its own issue.
+- The notebook-vs-`laya.train` epoch-seed formula is a reproducibility mismatch, also separate.
diff --git a/research/scripts/step4_benchmark/README.md b/research/scripts/step4_benchmark/README.md
index 27fc8000..9287e697 100644
--- a/research/scripts/step4_benchmark/README.md
+++ b/research/scripts/step4_benchmark/README.md
@@ -105,7 +105,9 @@ python research/scripts/step4_benchmark/run_benchmark.py --gate D \
--data data/typed_decisions/train.jsonl --out-dir runs/step4 --device cuda
```
-Summarise paired per-decision outcomes across seeds, not just the two means.
+Summarise paired per-decision outcomes across seeds, not just the two means. The measured
+three-seed panel and its verdict are in [`GATE_D_RESULTS.md`](GATE_D_RESULTS.md), with the raw cells
+under [`results/`](results/).
### 20-option order stability
diff --git a/research/scripts/step4_benchmark/results/step4_gate_d_results.json b/research/scripts/step4_benchmark/results/step4_gate_d_results.json
new file mode 100644
index 00000000..3eb8967e
--- /dev/null
+++ b/research/scripts/step4_benchmark/results/step4_gate_d_results.json
@@ -0,0 +1,308 @@
+{
+ "meta": {
+ "pinned_sha": "a4a8921afebfd852bba0000475cfb6ab737a124c",
+ "torch": "2.11.0+cu128",
+ "gpu": "Tesla T4",
+ "visible_gpus": 1,
+ "base_config": {
+ "max_len": 512,
+ "head_max_len": 192
+ },
+ "n_train_cases": 1200,
+ "n_test_cases": 400,
+ "reused": {
+ "rlcd_seed0": "Gate R kernel (separate session, same SHA/data/command)"
+ }
+ },
+ "rlcd_seed0": {
+ "checkpoint": "Gate R kernel run_rlcd_seed0 (reused)",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.775,
+ "ece": 0.1504,
+ "brier": 0.0519,
+ "soft_accuracy": 0.5654,
+ "per_type_accuracy": {
+ "choice": 0.7417,
+ "noul": 0.8517,
+ "score": 0.7425
+ },
+ "loss": "rlcd",
+ "seed": 0,
+ "wall_clock_s": 2691.3
+ },
+ "base": {
+ "checkpoint": "convaiinnovations/laya",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.362,
+ "ece": 0.1741,
+ "brier": 0.3287,
+ "soft_accuracy": 0.3656,
+ "score_mae": 0.6936,
+ "within_one_level": 0.7525,
+ "per_type_accuracy": {
+ "choice": 0.29,
+ "noul": 0.4867,
+ "score": 0.3225
+ },
+ "n_brier_decisions": 1200
+ },
+ "rlcd_seed1": {
+ "checkpoint": "/kaggle/working/run_rlcd_seed1",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.778,
+ "ece": 0.1477,
+ "brier": 0.0519,
+ "soft_accuracy": 0.5652,
+ "score_mae": 0.2155,
+ "within_one_level": 0.9888,
+ "per_type_accuracy": {
+ "choice": 0.74,
+ "noul": 0.8467,
+ "score": 0.755
+ },
+ "n_brier_decisions": 1200,
+ "loss": "rlcd",
+ "seed": 1,
+ "wall_clock_s": 2697.6,
+ "temperature": [
+ 1.0706707239151,
+ 1.0655159950256348,
+ 1.1516488790512085
+ ],
+ "calibration": {
+ "choice": {
+ "items": 141,
+ "temperature": 1.0706707239151,
+ "issues": [],
+ "accuracy": 0.7588652482269503,
+ "mean_confidence": 0.6076728429612874
+ },
+ "score": {
+ "items": 153,
+ "temperature": 1.0655159950256348,
+ "issues": [],
+ "accuracy": 0.7843137254901961,
+ "mean_confidence": 0.5692810567559323
+ },
+ "noul": {
+ "items": 106,
+ "temperature": 1.1516488790512085,
+ "issues": [],
+ "accuracy": 0.8584905660377359,
+ "mean_confidence": 0.7542289320023828
+ }
+ },
+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",
+ "finite": true
+ },
+ "soft-ce_seed0": {
+ "checkpoint": "/kaggle/working/run_soft_ce_seed0",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.7605,
+ "ece": 0.1353,
+ "brier": 0.0532,
+ "soft_accuracy": 0.5642,
+ "score_mae": 0.2456,
+ "within_one_level": 0.9938,
+ "per_type_accuracy": {
+ "choice": 0.7417,
+ "noul": 0.855,
+ "score": 0.7037
+ },
+ "n_brier_decisions": 1200,
+ "loss": "soft-ce",
+ "seed": 0,
+ "wall_clock_s": 2691.4,
+ "temperature": [
+ 1.0631763935089111,
+ 1.0451915264129639,
+ 1.0851869583129883
+ ],
+ "calibration": {
+ "choice": {
+ "items": 141,
+ "temperature": 1.0631763935089111,
+ "issues": [],
+ "accuracy": 0.7943262411347518,
+ "mean_confidence": 0.6069391803631538
+ },
+ "score": {
+ "items": 153,
+ "temperature": 1.0451915264129639,
+ "issues": [],
+ "accuracy": 0.7843137254901961,
+ "mean_confidence": 0.5572438980445797
+ },
+ "noul": {
+ "items": 106,
+ "temperature": 1.0851869583129883,
+ "issues": [],
+ "accuracy": 0.839622641509434,
+ "mean_confidence": 0.7526641343832678
+ }
+ },
+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",
+ "finite": true
+ },
+ "soft-ce_seed1": {
+ "checkpoint": "/kaggle/working/run_soft_ce_seed1",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.7915,
+ "ece": 0.1574,
+ "brier": 0.0481,
+ "soft_accuracy": 0.5705,
+ "score_mae": 0.2105,
+ "within_one_level": 0.9912,
+ "per_type_accuracy": {
+ "choice": 0.77,
+ "noul": 0.8583,
+ "score": 0.7575
+ },
+ "n_brier_decisions": 1200,
+ "loss": "soft-ce",
+ "seed": 1,
+ "wall_clock_s": 2705.1,
+ "temperature": [
+ 1.0398290157318115,
+ 1.0743035078048706,
+ 1.139175534248352
+ ],
+ "calibration": {
+ "choice": {
+ "items": 141,
+ "temperature": 1.0398290157318115,
+ "issues": [],
+ "accuracy": 0.8014184397163121,
+ "mean_confidence": 0.615891238130113
+ },
+ "score": {
+ "items": 153,
+ "temperature": 1.0743035078048706,
+ "issues": [],
+ "accuracy": 0.7777777777777778,
+ "mean_confidence": 0.5753281628764587
+ },
+ "noul": {
+ "items": 106,
+ "temperature": 1.139175534248352,
+ "issues": [],
+ "accuracy": 0.839622641509434,
+ "mean_confidence": 0.7543471859560601
+ }
+ },
+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",
+ "finite": true
+ },
+ "rlcd_seed2": {
+ "checkpoint": "/kaggle/working/run_rlcd_seed2",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.7755,
+ "ece": 0.153,
+ "brier": 0.049,
+ "soft_accuracy": 0.5638,
+ "score_mae": 0.2195,
+ "within_one_level": 0.9938,
+ "per_type_accuracy": {
+ "choice": 0.745,
+ "noul": 0.845,
+ "score": 0.7462
+ },
+ "n_brier_decisions": 1200,
+ "loss": "rlcd",
+ "seed": 2,
+ "wall_clock_s": 2708.5,
+ "temperature": [
+ 1.112676978111267,
+ 1.054789662361145,
+ 1.1128166913986206
+ ],
+ "calibration": {
+ "choice": {
+ "items": 141,
+ "temperature": 1.112676978111267,
+ "issues": [],
+ "accuracy": 0.7730496453900709,
+ "mean_confidence": 0.5887910391183622
+ },
+ "score": {
+ "items": 153,
+ "temperature": 1.054789662361145,
+ "issues": [],
+ "accuracy": 0.7908496732026143,
+ "mean_confidence": 0.5640255503811176
+ },
+ "noul": {
+ "items": 106,
+ "temperature": 1.1128166913986206,
+ "issues": [],
+ "accuracy": 0.8584905660377359,
+ "mean_confidence": 0.7526793880094345
+ }
+ },
+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",
+ "finite": true
+ },
+ "soft-ce_seed2": {
+ "checkpoint": "/kaggle/working/run_soft_ce_seed2",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.777,
+ "ece": 0.1471,
+ "brier": 0.0513,
+ "soft_accuracy": 0.5667,
+ "score_mae": 0.214,
+ "within_one_level": 0.99,
+ "per_type_accuracy": {
+ "choice": 0.75,
+ "noul": 0.8467,
+ "score": 0.745
+ },
+ "n_brier_decisions": 1200,
+ "loss": "soft-ce",
+ "seed": 2,
+ "wall_clock_s": 2701.8,
+ "temperature": [
+ 1.0387086868286133,
+ 1.053942084312439,
+ 1.08060884475708
+ ],
+ "calibration": {
+ "choice": {
+ "items": 141,
+ "temperature": 1.0387086868286133,
+ "issues": [],
+ "accuracy": 0.7730496453900709,
+ "mean_confidence": 0.6111094674737355
+ },
+ "score": {
+ "items": 153,
+ "temperature": 1.053942084312439,
+ "issues": [],
+ "accuracy": 0.8300653594771242,
+ "mean_confidence": 0.5717240346959584
+ },
+ "noul": {
+ "items": 106,
+ "temperature": 1.08060884475708,
+ "issues": [],
+ "accuracy": 0.8584905660377359,
+ "mean_confidence": 0.7583364282950572
+ }
+ },
+ "split_line": "train items 5600, calibration items 400, skipped {}, device cuda",
+ "finite": true
+ }
+}
\ No newline at end of file
diff --git a/research/scripts/step4_benchmark/results/step4_gate_r_results.json b/research/scripts/step4_benchmark/results/step4_gate_r_results.json
new file mode 100644
index 00000000..96c55a19
--- /dev/null
+++ b/research/scripts/step4_benchmark/results/step4_gate_r_results.json
@@ -0,0 +1,71 @@
+{
+ "meta": {
+ "pinned_sha": "a4a8921afebfd852bba0000475cfb6ab737a124c",
+ "torch": "2.11.0+cu128",
+ "gpu": "Tesla T4",
+ "visible_gpus": 1,
+ "train_seconds": 2691.3,
+ "peak_gpu_gb": 0.0,
+ "train_split_line": "train items 5600, calibration items 400, skipped {}, device cuda",
+ "base_config": {
+ "max_len": 512,
+ "head_max_len": 192
+ },
+ "n_train_cases": 1200,
+ "n_test_cases": 400
+ },
+ "rlcd_seed0": {
+ "checkpoint": "/kaggle/working/run_rlcd_seed0",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.775,
+ "ece": 0.1504,
+ "brier": 0.0519,
+ "soft_accuracy": 0.5654,
+ "score_mae": 0.2246,
+ "within_one_level": 0.99,
+ "per_type_accuracy": {
+ "choice": 0.7417,
+ "noul": 0.8517,
+ "score": 0.7425
+ },
+ "n_brier_decisions": 1200
+ },
+ "base": {
+ "checkpoint": "convaiinnovations/laya",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.362,
+ "ece": 0.1741,
+ "brier": 0.3287,
+ "soft_accuracy": 0.3656,
+ "score_mae": 0.6936,
+ "within_one_level": 0.7525,
+ "per_type_accuracy": {
+ "choice": 0.29,
+ "noul": 0.4867,
+ "score": 0.3225
+ },
+ "n_brier_decisions": 1200
+ },
+ "published": {
+ "checkpoint": "convaiinnovations/laya-typed-decisions",
+ "temperatures": "shipped",
+ "n_cases": 400,
+ "n_decisions": 2000,
+ "accuracy": 0.7665,
+ "ece": 0.2137,
+ "brier": 0.0658,
+ "soft_accuracy": 0.5087,
+ "score_mae": 0.2424,
+ "within_one_level": 0.995,
+ "per_type_accuracy": {
+ "choice": 0.7333,
+ "noul": 0.8567,
+ "score": 0.7238
+ },
+ "n_brier_decisions": 1200
+ }
+}
\ No newline at end of file