Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
updated baseline results
Noah Cylich8648e0e697ab
diff --git a/docs/matryoshka_results.md b/docs/matryoshka_results.mdindex b88a3ba..0b22635 100644--- a/docs/matryoshka_results.md+++ b/docs/matryoshka_results.md@@ -1,6 +1,4 @@-# FFN Interior Matryoshka — Results--> **Note**: These results predate the mixer refactor. At the time, matryoshka FFN masking was only applied to attention FFNs and the local FFN in encoder blocks — the MLP-Mixer token/channel mixing layers were **not** masked. Current code applies masking to all FFN layers including mixer.+# Matryoshka — Results## Approach@@ -11,9 +9,13 @@ Heterogeneous batch matryoshka: a per-batch-item prefix mask zeros out FFN neuro- **4x**: d_ff = 512 (first 512 neurons)- **8x**: d_ff = 256 (first 256 neurons)-d_model, attention heads, embeddings, and mixer are unchanged across all widths.+d_model, attention heads, and embeddings are unchanged across all widths.++Two masking scopes:+- **FFN-only**: masks decoder FFN and encoder local FFN. MLP-Mixer token/channel mixing layers are NOT masked.+- **Full matryoshka** (current): masks ALL FFN layers including MLP-Mixer token/channel mixing.-Two modes:+Two input modes:- **Unique input** (default): each batch item is a unique sample, assigned a fixed width based on its position in the batch. Full data diversity per epoch.- **Shared input** (`--mat-shared-input`): each unique sample is repeated across all widths. Smaller unique batch but every sample trains every width. Epoch = 4x more steps to see all data.@@ -23,7 +25,6 @@ All runs use the same architecture and hyperparameters:| Setting | Value ||---|---|-| Parameters | 87,666,176 || d_model | 512 || d_ff | 2048 || Heads (KV) | 8 (8) |@@ -40,65 +41,61 @@ All runs use the same architecture and hyperparameters:## Results (1 epoch, val PPL)-| Model | Main (output-only slice) | FFN unique input | FFN shared input |-|---|---|---|---|-| Full (d_ff=2048) | **4.52** | 4.66 | 5.11 |-| 2x (d_ff=1024) | - | **4.66** | 5.11 |-| 4x (d_ff=512) | - | **4.68** | 5.11 |-| 8x (d_ff=256) | - | **4.80** | 5.20 |-| Quant (INT4 g32) | **4.59** | 4.75 | 5.20 |--> Main branch PPLs omitted — output-only slice does not constrain internal computation, so sub-model PPLs are misleading.--| Metric | Main | FFN unique | FFN shared |-|---|---|---|---|-| Mat method | Output-only slice | FFN interior mask | FFN interior mask |-| Total steps | 11,038 | 11,038 | 44,156 |-| Unique items/batch | 32/dev | 32/dev | 8/dev x 4 |-| Speed | ~19.3 it/s | 11.8 it/s | 20.9 it/s |-| Wall time | ~10 min | ~16 min | ~36 min |-| Avg train loss | 4.43 | 4.42 | 3.12 |-| Final train loss | 1.51 | 1.53 | 1.66 |-| Text val PPL | **4.52** | 4.66 | 5.11 |-| Quant val PPL | **4.59** | 4.75 | 5.20 |-| Weight sparsity | 50.02% | 50.01% | 68.28% |-| Speech val PPL | 87.26 | 92.59 | 78.64 |-| Speech WER | 1.0000 | 0.9695 | 0.9618 |-| Throughput | 135.7 tok/s | 46.3 tok/s | 156.8 tok/s |+| Model | No mat (main) | FFN-only | FFN-only shared | **Full mat (FFN + mixer)** |+|---|---|---|---|---|+| Full (d_ff=2048) | **4.52** | 4.66 | 5.11 | 4.82 |+| 2x (d_ff=1024) | - | 4.66 | 5.11 | 4.82 |+| 4x (d_ff=512) | - | 4.68 | 5.11 | 4.85 |+| 8x (d_ff=256) | - | **4.80** | 5.20 | 4.97 |+| Quant (INT4 g32) | **4.59** | 4.75 | 5.20 | 4.94 |++> No-mat sub-model PPLs omitted — output-only slice does not constrain internal computation, so sub-model PPLs are misleading.++| Metric | No mat | FFN-only | FFN-only shared | Full mat |+|---|---|---|---|---|+| Masking scope | Output-only slice | Decoder+encoder FFN | Decoder+encoder FFN | All FFN + mixer |+| Parameters | 87,666,176 | 87,666,176 | 87,666,176 | 88,535,040 |+| Total steps | 11,038 | 11,038 | 44,156 | 11,038 |+| Unique items/batch | 32/dev | 32/dev | 8/dev x 4 | 32/dev |+| Speed | ~19.3 it/s | 11.8 it/s | 20.9 it/s | 10.2 it/s |+| Wall time | ~10 min | ~16 min | ~36 min | ~18 min |+| Avg train loss | 4.43 | 4.42 | 3.12 | - |+| Final train loss | 1.51 | 1.53 | 1.66 | - |+| Text val PPL | **4.52** | 4.66 | 5.11 | 4.82 |+| Quant val PPL | **4.59** | 4.75 | 5.20 | 4.94 |+| Weight sparsity | 50.02% | 50.01% | 68.28% | 49.52% |+| Speech val PPL | 87.26 | 92.59 | 78.64 | 83.56 |+| Speech WER | 1.0000 | 0.9695 | 0.9618 | 0.9924 |+| Throughput | 135.7 tok/s | 46.3 tok/s | 156.8 tok/s | 146.6 tok/s |## Analysis-**Main branch full-model PPL (4.52) is the best**, as expected — it doesn't spend any capacity on sub-model training. FFN unique input (4.66) pays a +0.14 PPL cost for genuinely training 4 widths simultaneously.+**No-mat full-model PPL (4.52) is the best**, as expected — it doesn't spend any capacity on sub-model training.-**FFN unique input beats shared input** (4.66 vs 5.11). Both see the same total unique data per epoch, but unique input has 4x data diversity per step (32 unique items vs 8 repeated 4x).+**FFN-only (4.66) is the best matryoshka approach** for text PPL — it pays +0.14 over no-mat for genuinely training 4 widths simultaneously, with only a +0.14 gap from full to 8x (4.66→4.80).-**Shared input over-prunes** (68% vs 50% target) because the pruning schedule fractions apply to 4x more total steps, creating a wider absolute pruning window.+**Full matryoshka (4.82) pays an additional +0.16 over FFN-only** for also masking the mixer layers. The sub-model spread is similarly tight (full→8x: 0.15 PPL). The extra cost buys truly independent sub-models where even the mixer's token and channel mixing operate at reduced width.-**FFN interior matryoshka sub-models are tight.** With unique input, the d_ff=256 sub-model (8x, 68.7M params) is only +0.14 PPL worse than full. This is a genuine sub-network — unlike main's output-only approach which shares all internal computation.+**Shared input is not worth it** (4.66 vs 5.11 for FFN-only). Unique input with random width assignment per item is sufficient — every sample doesn't need to train every width.-**Main is faster** (~19 it/s) because it runs one full forward + cheap logit slicing. FFN unique input (11.8 it/s) does one forward with heterogeneous masking but can't exploit XLA batch optimizations as well. FFN shared input (20.9 it/s) is fastest per-step due to repeated inputs.+**Full mat is slower** (10.2 it/s vs 11.8 FFN-only) because the mixer FFNs are also heterogeneously masked, adding compute overhead.### Recommendation-Use **unique input** (default, no flag) for best quality. The +0.14 PPL cost over main buys genuinely independent sub-models that can be exported at reduced FFN width.+Use **unique input** (default, no flag) for best quality. Choose between FFN-only and full matryoshka based on whether mixer-level sub-models are needed:+- **FFN-only**: better text PPL (4.66 vs 4.82), faster (11.8 vs 10.2 it/s). Sub-models share full mixer capacity.+- **Full matryoshka**: truly independent sub-models at every layer. Slightly worse but sub-models are more self-contained for export.## Training Speed| Approach | it/s | Notes ||---|---|---|-| Main branch (output-only slice) | ~19.3 | 1 forward + logit slicing (no true sub-models) |-| FFN unique input | 11.8 | Single forward, heterogeneous batch |-| FFN shared input | 20.9 | Single forward, repeated inputs (XLA optimizes) |--## Generation Samples (Unique Input)--```-<story>[Once upon a time] where gan the vessel was calculated with a sensible light-<story>[The little dog] and the big boots and buntline furnished with a very little red buttons and si-<story>[She was very happy because] but where if you ever see her again i know she has a very special thing i'm go-```+| No mat (output-only slice) | ~19.3 | 1 forward + logit slicing (no true sub-models) |+| FFN-only unique input | 11.8 | Single forward, heterogeneous batch (decoder+encoder FFN) |+| FFN-only shared input | 20.9 | Single forward, repeated inputs (XLA optimizes) |+| Full mat unique input | 10.2 | Single forward, heterogeneous batch (all FFN + mixer) |-## Ablation: Shared Input at 4x Batch Size+## Ablation: Shared Input at 4x Batch Size (FFN-only)**Question**: Does training every sample at all widths help, if we control for gradient quality?@@ -115,7 +112,7 @@ With `--mat-shared-input --batch-size 128`, each step has 32 unique samples repe| Wall time | ~16 min | ~35 min || Sparsity | 50.01% | 50.01% |-**Result**: Essentially identical PPL (4.66 vs 4.67). Repeating inputs across widths provides no benefit when gradient quality (unique samples per step) is controlled. Random width assignment per item is sufficient — the model doesn't need every sample to train every width simultaneously.+**Result**: Essentially identical PPL (4.66 vs 4.67). Repeating inputs across widths provides no benefit when gradient quality (unique samples per step) is controlled. Random width assignment per item is sufficient.This confirms **unique input at batch=32 is optimal**: same quality, 2x faster.