Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

updated baseline results

Noah Cylich8648e0e697ab
diff --git a/docs/matryoshka_results.md b/docs/matryoshka_results.md
index b88a3ba..0b22635 100644
--- a/docs/matryoshka_results.md
+++ b/docs/matryoshka_results.md
@@ -1,6 +1,4 @@
-# FFN Interior Matryoshka — Results
-
-> **Note**: These results predate the mixer refactor. At the time, matryoshka FFN masking was only applied to attention FFNs and the local FFN in encoder blocks — the MLP-Mixer token/channel mixing layers were **not** masked. Current code applies masking to all FFN layers including mixer.
+# Matryoshka — Results
## Approach
@@ -11,9 +9,13 @@ Heterogeneous batch matryoshka: a per-batch-item prefix mask zeros out FFN neuro
- **4x**: d_ff = 512 (first 512 neurons)
- **8x**: d_ff = 256 (first 256 neurons)
-d_model, attention heads, embeddings, and mixer are unchanged across all widths.
+d_model, attention heads, and embeddings are unchanged across all widths.
+
+Two masking scopes:
+- **FFN-only**: masks decoder FFN and encoder local FFN. MLP-Mixer token/channel mixing layers are NOT masked.
+- **Full matryoshka** (current): masks ALL FFN layers including MLP-Mixer token/channel mixing.
-Two modes:
+Two input modes:
- **Unique input** (default): each batch item is a unique sample, assigned a fixed width based on its position in the batch. Full data diversity per epoch.
- **Shared input** (`--mat-shared-input`): each unique sample is repeated across all widths. Smaller unique batch but every sample trains every width. Epoch = 4x more steps to see all data.
@@ -23,7 +25,6 @@ All runs use the same architecture and hyperparameters:
| Setting | Value |
|---|---|
-| Parameters | 87,666,176 |
| d_model | 512 |
| d_ff | 2048 |
| Heads (KV) | 8 (8) |
@@ -40,65 +41,61 @@ All runs use the same architecture and hyperparameters:
## Results (1 epoch, val PPL)
-| Model | Main (output-only slice) | FFN unique input | FFN shared input |
-|---|---|---|---|
-| Full (d_ff=2048) | **4.52** | 4.66 | 5.11 |
-| 2x (d_ff=1024) | - | **4.66** | 5.11 |
-| 4x (d_ff=512) | - | **4.68** | 5.11 |
-| 8x (d_ff=256) | - | **4.80** | 5.20 |
-| Quant (INT4 g32) | **4.59** | 4.75 | 5.20 |
-
-> Main branch PPLs omitted — output-only slice does not constrain internal computation, so sub-model PPLs are misleading.
-
-| Metric | Main | FFN unique | FFN shared |
-|---|---|---|---|
-| Mat method | Output-only slice | FFN interior mask | FFN interior mask |
-| Total steps | 11,038 | 11,038 | 44,156 |
-| Unique items/batch | 32/dev | 32/dev | 8/dev x 4 |
-| Speed | ~19.3 it/s | 11.8 it/s | 20.9 it/s |
-| Wall time | ~10 min | ~16 min | ~36 min |
-| Avg train loss | 4.43 | 4.42 | 3.12 |
-| Final train loss | 1.51 | 1.53 | 1.66 |
-| Text val PPL | **4.52** | 4.66 | 5.11 |
-| Quant val PPL | **4.59** | 4.75 | 5.20 |
-| Weight sparsity | 50.02% | 50.01% | 68.28% |
-| Speech val PPL | 87.26 | 92.59 | 78.64 |
-| Speech WER | 1.0000 | 0.9695 | 0.9618 |
-| Throughput | 135.7 tok/s | 46.3 tok/s | 156.8 tok/s |
+| Model | No mat (main) | FFN-only | FFN-only shared | **Full mat (FFN + mixer)** |
+|---|---|---|---|---|
+| Full (d_ff=2048) | **4.52** | 4.66 | 5.11 | 4.82 |
+| 2x (d_ff=1024) | - | 4.66 | 5.11 | 4.82 |
+| 4x (d_ff=512) | - | 4.68 | 5.11 | 4.85 |
+| 8x (d_ff=256) | - | **4.80** | 5.20 | 4.97 |
+| Quant (INT4 g32) | **4.59** | 4.75 | 5.20 | 4.94 |
+
+> No-mat sub-model PPLs omitted — output-only slice does not constrain internal computation, so sub-model PPLs are misleading.
+
+| Metric | No mat | FFN-only | FFN-only shared | Full mat |
+|---|---|---|---|---|
+| Masking scope | Output-only slice | Decoder+encoder FFN | Decoder+encoder FFN | All FFN + mixer |
+| Parameters | 87,666,176 | 87,666,176 | 87,666,176 | 88,535,040 |
+| Total steps | 11,038 | 11,038 | 44,156 | 11,038 |
+| Unique items/batch | 32/dev | 32/dev | 8/dev x 4 | 32/dev |
+| Speed | ~19.3 it/s | 11.8 it/s | 20.9 it/s | 10.2 it/s |
+| Wall time | ~10 min | ~16 min | ~36 min | ~18 min |
+| Avg train loss | 4.43 | 4.42 | 3.12 | - |
+| Final train loss | 1.51 | 1.53 | 1.66 | - |
+| Text val PPL | **4.52** | 4.66 | 5.11 | 4.82 |
+| Quant val PPL | **4.59** | 4.75 | 5.20 | 4.94 |
+| Weight sparsity | 50.02% | 50.01% | 68.28% | 49.52% |
+| Speech val PPL | 87.26 | 92.59 | 78.64 | 83.56 |
+| Speech WER | 1.0000 | 0.9695 | 0.9618 | 0.9924 |
+| Throughput | 135.7 tok/s | 46.3 tok/s | 156.8 tok/s | 146.6 tok/s |
## Analysis
-**Main branch full-model PPL (4.52) is the best**, as expected — it doesn't spend any capacity on sub-model training. FFN unique input (4.66) pays a +0.14 PPL cost for genuinely training 4 widths simultaneously.
+**No-mat full-model PPL (4.52) is the best**, as expected — it doesn't spend any capacity on sub-model training.
-**FFN unique input beats shared input** (4.66 vs 5.11). Both see the same total unique data per epoch, but unique input has 4x data diversity per step (32 unique items vs 8 repeated 4x).
+**FFN-only (4.66) is the best matryoshka approach** for text PPL — it pays +0.14 over no-mat for genuinely training 4 widths simultaneously, with only a +0.14 gap from full to 8x (4.66→4.80).
-**Shared input over-prunes** (68% vs 50% target) because the pruning schedule fractions apply to 4x more total steps, creating a wider absolute pruning window.
+**Full matryoshka (4.82) pays an additional +0.16 over FFN-only** for also masking the mixer layers. The sub-model spread is similarly tight (full→8x: 0.15 PPL). The extra cost buys truly independent sub-models where even the mixer's token and channel mixing operate at reduced width.
-**FFN interior matryoshka sub-models are tight.** With unique input, the d_ff=256 sub-model (8x, 68.7M params) is only +0.14 PPL worse than full. This is a genuine sub-network — unlike main's output-only approach which shares all internal computation.
+**Shared input is not worth it** (4.66 vs 5.11 for FFN-only). Unique input with random width assignment per item is sufficient — every sample doesn't need to train every width.
-**Main is faster** (~19 it/s) because it runs one full forward + cheap logit slicing. FFN unique input (11.8 it/s) does one forward with heterogeneous masking but can't exploit XLA batch optimizations as well. FFN shared input (20.9 it/s) is fastest per-step due to repeated inputs.
+**Full mat is slower** (10.2 it/s vs 11.8 FFN-only) because the mixer FFNs are also heterogeneously masked, adding compute overhead.
### Recommendation
-Use **unique input** (default, no flag) for best quality. The +0.14 PPL cost over main buys genuinely independent sub-models that can be exported at reduced FFN width.
+Use **unique input** (default, no flag) for best quality. Choose between FFN-only and full matryoshka based on whether mixer-level sub-models are needed:
+- **FFN-only**: better text PPL (4.66 vs 4.82), faster (11.8 vs 10.2 it/s). Sub-models share full mixer capacity.
+- **Full matryoshka**: truly independent sub-models at every layer. Slightly worse but sub-models are more self-contained for export.
## Training Speed
| Approach | it/s | Notes |
|---|---|---|
-| Main branch (output-only slice) | ~19.3 | 1 forward + logit slicing (no true sub-models) |
-| FFN unique input | 11.8 | Single forward, heterogeneous batch |
-| FFN shared input | 20.9 | Single forward, repeated inputs (XLA optimizes) |
-
-## Generation Samples (Unique Input)
-
-```
-<story>[Once upon a time] where gan the vessel was calculated with a sensible light
-<story>[The little dog] and the big boots and buntline furnished with a very little red buttons and si
-<story>[She was very happy because] but where if you ever see her again i know she has a very special thing i'm go
-```
+| No mat (output-only slice) | ~19.3 | 1 forward + logit slicing (no true sub-models) |
+| FFN-only unique input | 11.8 | Single forward, heterogeneous batch (decoder+encoder FFN) |
+| FFN-only shared input | 20.9 | Single forward, repeated inputs (XLA optimizes) |
+| Full mat unique input | 10.2 | Single forward, heterogeneous batch (all FFN + mixer) |
-## Ablation: Shared Input at 4x Batch Size
+## Ablation: Shared Input at 4x Batch Size (FFN-only)
**Question**: Does training every sample at all widths help, if we control for gradient quality?
@@ -115,7 +112,7 @@ With `--mat-shared-input --batch-size 128`, each step has 32 unique samples repe
| Wall time | ~16 min | ~35 min |
| Sparsity | 50.01% | 50.01% |
-**Result**: Essentially identical PPL (4.66 vs 4.67). Repeating inputs across widths provides no benefit when gradient quality (unique samples per step) is controlled. Random width assignment per item is sufficient — the model doesn't need every sample to train every width simultaneously.
+**Result**: Essentially identical PPL (4.66 vs 4.67). Repeating inputs across widths provides no benefit when gradient quality (unique samples per step) is controlled. Random width assignment per item is sufficient.
This confirms **unique input at batch=32 is optimal**: same quality, 2x faster.