Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
proper matryoshka and results
Noah Cylichcbc38056e69f
diff --git a/docs/matryoshka_results.md b/docs/matryoshka_results.mdnew file mode 100644index 0000000..8ee8627--- /dev/null+++ b/docs/matryoshka_results.md@@ -0,0 +1,103 @@+# FFN Interior Matryoshka — Results++## Approach++Heterogeneous batch MRL: a per-batch-item prefix mask zeros out FFN neurons beyond the target width. Single forward-backward pass, single optimizer step. All MRL widths trained simultaneously.++- **Full width**: d_ff = 2048 (all neurons active)+- **MRL 256**: d_ff = 1024 (first 1024 neurons)+- **MRL 128**: d_ff = 512 (first 512 neurons)+- **MRL 64**: d_ff = 256 (first 256 neurons)++d_model, attention heads, embeddings, and mixer are unchanged across all widths.++Two modes:+- **Unique input** (default): each batch item is a unique sample, assigned a fixed width based on its position in the batch. Full data diversity per epoch.+- **Shared input** (`--mrl-shared-input`): each unique sample is repeated across all widths. Smaller unique batch but every sample trains every width. Epoch = 4x more steps to see all data.++## Training Config++All runs use the same architecture and hyperparameters:++| Setting | Value |+|---|---|+| Parameters | 87,666,176 |+| d_model | 512 |+| d_ff | 2048 |+| Heads (KV) | 8 (8) |+| Layers | 8 enc / 4 dec |+| Memory slots | 64 |+| Activation | dReLU |+| Dtype | bfloat16 |+| Devices | 8 (TPU v6e-8) |+| Batch | 32 x 8 = 256 |+| LR (Adam) | 0.0003 x 8 = 0.0024 |+| LR (Muon) | 0.02 -> 0.0566 |+| Sparsity | 50% target (gradual cubic, group=32) |+| Speech | every 3 text steps |++## Results (1 epoch, val PPL)++| Model | Main (output-only slice) | FFN unique input | FFN shared input |+|---|---|---|---|+| Full (d_ff=2048) | **4.52** | 4.66 | 5.11 |+| MRL 256 (d_ff=1024) | - | **4.66** | 5.11 |+| MRL 128 (d_ff=512) | - | **4.68** | 5.11 |+| MRL 64 (d_ff=256) | - | **4.80** | 5.20 |+| Quant (INT4 g32) | **4.59** | 4.75 | 5.20 |++> Main branch MRL PPLs omitted — output-only slice MRL does not constrain internal computation, so sub-model PPLs are misleading.++| Metric | Main | FFN unique | FFN shared |+|---|---|---|---|+| MRL method | Output-only slice | FFN interior mask | FFN interior mask |+| Total steps | 11,038 | 11,038 | 44,156 |+| Unique items/batch | 32/dev | 32/dev | 8/dev x 4 |+| Speed | ~19.3 it/s | 11.8 it/s | 20.9 it/s |+| Wall time | ~10 min | ~16 min | ~36 min |+| Avg train loss | 4.43 | 4.42 | 3.12 |+| Final train loss | 1.51 | 1.53 | 1.66 |+| Text val PPL | **4.52** | 4.66 | 5.11 |+| Quant val PPL | **4.59** | 4.75 | 5.20 |+| Weight sparsity | 50.02% | 50.01% | 68.28% |+| Speech val PPL | 87.26 | 92.59 | 78.64 |+| Speech WER | 1.0000 | 0.9695 | 0.9618 |+| Throughput | 135.7 tok/s | 46.3 tok/s | 156.8 tok/s |++## Analysis++**Main branch full-model PPL (4.52) is the best**, as expected — it doesn't spend any capacity on sub-model training. FFN unique input (4.66) pays a +0.14 PPL cost for genuinely training 4 widths simultaneously.++**FFN unique input beats shared input** (4.66 vs 5.11). Both see the same total unique data per epoch, but unique input has 4x data diversity per step (32 unique items vs 8 repeated 4x).++**Shared input over-prunes** (68% vs 50% target) because the pruning schedule fractions apply to 4x more total steps, creating a wider absolute pruning window.++**FFN interior MRL sub-models are tight.** With unique input, the d_ff=256 sub-model (1/8 of FFN, 68.7M params) is only +0.14 PPL worse than full. This is a genuine sub-network — unlike main's output-only approach which shares all internal computation.++**Main is faster** (~19 it/s) because it runs one full forward + cheap logit slicing. FFN unique input (11.8 it/s) does one forward with heterogeneous masking but can't exploit XLA batch optimizations as well. FFN shared input (20.9 it/s) is fastest per-step due to repeated inputs.++### Recommendation++Use **unique input** (default, no flag) for best quality. The +0.14 PPL cost over main buys genuinely independent sub-models that can be exported at reduced FFN width.++## Training Speed++| Approach | it/s | Notes |+|---|---|---|+| Main branch (output-only slice) | ~19.3 | 1 forward + logit slicing (no true sub-models) |+| FFN unique input | 11.8 | Single forward, heterogeneous batch |+| FFN shared input | 20.9 | Single forward, repeated inputs (XLA optimizes) |++## Generation Samples (Unique Input)++```+<story>[Once upon a time] where gan the vessel was calculated with a sensible light+<story>[The little dog] and the big boots and buntline furnished with a very little red buttons and si+<story>[She was very happy because] but where if you ever see her again i know she has a very special thing i'm go+```++## Notes++- All runs are 1-epoch on TinyStories + LibriSpeech. Multi-epoch runs would improve all metrics.+- Speech WER is poor across all methods — expected with interleaved training and limited speech data.+- The heterogeneous batch approach uses the same memory as a single full-width forward (~1x). No overhead from MRL.diff --git a/src/cli.py b/src/cli.pyindex e912af0..93bef0c 100644--- a/src/cli.py+++ b/src/cli.py@@ -120,6 +120,8 @@ def main():p.add_argument("--num-memory-slots", type=int, default=64)p.add_argument("--mrl-dims", type=int, nargs="*", default=[256, 128, 64],help="MRL dimension targets (default: 256 128 64)")+ p.add_argument("--mrl-shared-input", action="store_true",+ help="Each unique input is repeated across all MRL widths (default: unique input per width)")p.add_argument("--no-speech", action="store_true", help="Disable speech training (text-only)")p.add_argument("--speech-every", type=int, default=3,help="Do one speech step every N text steps (default: 3)")diff --git a/src/model.py b/src/model.pyindex 38b0b2a..d8c27e9 100644--- a/src/model.py+++ b/src/model.py@@ -129,31 +129,17 @@ class FeedForward(nn.Module):activation: str = "drelu"@nn.compact- def __call__(self, x, d_ff_slice=None):- sliced = d_ff_slice is not None and d_ff_slice < self.d_ff-- if sliced:- # Sliced path: access existing kernels and slice for efficiency.- # Params must already exist from a prior full-width call (e.g. the- # non-MRL forward in forward_with_aux).- gk = self.variables['params']['gate_proj']['kernel'][:, :d_ff_slice]- uk = self.variables['params']['up_proj']['kernel'][:, :d_ff_slice]- gate = x.astype(self.dtype) @ gk.astype(self.dtype)- up = x.astype(self.dtype) @ uk.astype(self.dtype)- else:- gate = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="gate_proj")(x)- up = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="up_proj")(x)-+ def __call__(self, x, ffn_mask=None):+ gate = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="gate_proj")(x)+ up = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="up_proj")(x)if self.activation == "swiglu":h = nn.silu(gate) * upelif self.activation == "geglu":h = nn.gelu(gate) * upelse: # dreluh = nn.relu(gate) * nn.relu(up)-- if sliced:- dk = self.variables['params']['down_proj']['kernel'][:d_ff_slice]- return (h @ dk.astype(self.dtype))+ if ffn_mask is not None:+ h = h * ffn_mask[:, None, :] # (batch, 1, d_ff)return nn.Dense(self.d_model, dtype=self.dtype, use_bias=False, kernel_init=residual_init(self.num_layers), name="down_proj")(h)@@ -210,7 +196,7 @@ class MemoryMixerBlock(nn.Module):activation: str = "drelu"@nn.compact- def __call__(self, x, s, mask=None, rope=None, d_ff_slice=None):+ def __call__(self, x, s, mask=None, rope=None, ffn_mask=None):s_norm = ZCRMSNorm(dtype=self.dtype, name="pack_s_norm")(s)x_norm = ZCRMSNorm(dtype=self.dtype, name="pack_x_norm")(x)s = s + MultiHeadAttention(@@ -224,7 +210,7 @@ class MemoryMixerBlock(nn.Module):residual = xx = ZCRMSNorm(dtype=self.dtype, name="local_norm")(x)- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation, name="local_ffn")(x, d_ff_slice=d_ff_slice)+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation, name="local_ffn")(x, ffn_mask=ffn_mask)x = x + residualreturn x, s@@ -235,7 +221,7 @@ class MemoryMixerEncoder(nn.Module):config: TransformerConfig@nn.compact- def __call__(self, x, mask=None, rope=None, d_ff_slice=None):+ def __call__(self, x, mask=None, rope=None, ffn_mask=None):cfg = self.configdt = cfg.jax_dtypex = x.astype(dt)@@ -262,7 +248,7 @@ class MemoryMixerEncoder(nn.Module):x, s = nn.remat(MemoryMixerBlock)(cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff,cfg.num_memory_slots, cfg.total_layers, dt, cfg.activation, name=f"block_{i}"- )(x, s, mask=mask, rope=rope, d_ff_slice=d_ff_slice)+ )(x, s, mask=mask, rope=rope, ffn_mask=ffn_mask)s = ZCRMSNorm(dtype=dt, name="final_norm")(s)return s@@ -278,7 +264,7 @@ class DecoderBlock(nn.Module):activation: str = "drelu"@nn.compact- def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, d_ff_slice=None):+ def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, ffn_mask=None):residual = xx = ZCRMSNorm(dtype=self.dtype)(x)x = MultiHeadAttention(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers, self.dtype, name="self_attn")(@@ -295,7 +281,7 @@ class DecoderBlock(nn.Module):residual = xx = ZCRMSNorm(dtype=self.dtype)(x)- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, d_ff_slice=d_ff_slice)+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)x = x + residualreturn x@@ -306,7 +292,7 @@ class Decoder(nn.Module):config: TransformerConfig@nn.compact- def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, d_ff_slice=None):+ def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, ffn_mask=None):cfg = self.configdt = cfg.jax_dtypex = x.astype(dt)@@ -314,7 +300,7 @@ class Decoder(nn.Module):for i in range(cfg.num_decoder_layers):x = nn.remat(DecoderBlock)(cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, dt, cfg.activation, name=f"block_{i}"- )(x, encoder_out, self_mask=self_mask, cross_mask=cross_mask, rope=rope, d_ff_slice=d_ff_slice)+ )(x, encoder_out, self_mask=self_mask, cross_mask=cross_mask, rope=rope, ffn_mask=ffn_mask)x = ZCRMSNorm(dtype=dt)(x)return x@@ -346,19 +332,19 @@ class EncoderDecoderTransformer(nn.Module):head_dim = self.config.d_model // self.config.num_headsreturn precompute_rope_freqs(head_dim, seq_len, self.config.rope_theta)- def encode_text(self, src, src_mask=None, d_ff_slice=None):+ def encode_text(self, src, src_mask=None, ffn_mask=None):x = self.embedding(src) * self.embed_scalerope = self._rope(src.shape[1])- return self.encoder(x, mask=src_mask, rope=rope, d_ff_slice=d_ff_slice)+ return self.encoder(x, mask=src_mask, rope=rope, ffn_mask=ffn_mask)def encode(self, src, src_mask=None):"""Backward-compatible alias for encode_text."""return self.encode_text(src, src_mask=src_mask)- def encode_speech(self, mel, src_mask=None, d_ff_slice=None):+ def encode_speech(self, mel, src_mask=None, ffn_mask=None):x = self.mel_proj(mel) * self.embed_scalerope = self._rope(mel.shape[1])- return self.encoder(x, mask=src_mask, rope=rope, d_ff_slice=d_ff_slice)+ return self.encoder(x, mask=src_mask, rope=rope, ffn_mask=ffn_mask)def decode(self, tgt, encoder_out, self_mask=None, cross_mask=None):x = self.embedding(tgt) * self.embed_scale@@ -376,11 +362,11 @@ class EncoderDecoderTransformer(nn.Module):)return logits- def _run_decoder(self, encoder_out, tgt, tgt_mask=None, d_ff_slice=None):+ def _run_decoder(self, encoder_out, tgt, tgt_mask=None, ffn_mask=None):"""Run decoder and return float32 hidden states."""x = self.embedding(tgt) * self.embed_scalerope = self._rope(tgt.shape[1])- x = self.decoder(x, encoder_out, self_mask=tgt_mask, cross_mask=None, rope=rope, d_ff_slice=d_ff_slice)+ x = self.decoder(x, encoder_out, self_mask=tgt_mask, cross_mask=None, rope=rope, ffn_mask=ffn_mask)return x.astype(jnp.float32)def _slot_diversity(self, encoder_out):@@ -389,10 +375,29 @@ class EncoderDecoderTransformer(nn.Module):diag_sq = jnp.sum(jnp.diagonal(gram, axis1=1, axis2=2) ** 2)return (jnp.sum(gram ** 2) - diag_sq) / s.shape[0]+ def forward_masked(self, src, tgt, src_mask=None, tgt_mask=None, ffn_mask=None):+ """Single forward with per-batch-item FFN masking. Returns (logits, slot_div)."""+ encoder_out = self.encode_text(src, src_mask=src_mask, ffn_mask=ffn_mask)+ x_f32 = self._run_decoder(encoder_out, tgt, tgt_mask=tgt_mask, ffn_mask=ffn_mask)+ logits = x_f32 @ self.embedding.embedding.T+ slot_div = self._slot_diversity(encoder_out)+ return logits, slot_div++ def forward_speech_masked(self, mel, tgt, src_mask=None, tgt_mask=None, ffn_mask=None):+ """Single speech forward with per-batch-item FFN masking. Returns (logits, slot_div)."""+ encoder_out = self.encode_speech(mel, src_mask=src_mask, ffn_mask=ffn_mask)+ x_f32 = self._run_decoder(encoder_out, tgt, tgt_mask=tgt_mask, ffn_mask=ffn_mask)+ logits = x_f32 @ self.embedding.embedding.T+ slot_div = self._slot_diversity(encoder_out)+ return logits, slot_div+def forward_with_aux(self, src, tgt, src_mask=None, tgt_mask=None, cross_mask=None, mrl_dims=None):+ """Eval-only: separate per-width forwards for reporting per-width PPL."""emb = self.embedding.embedding+ d_ff = self.config.d_ff+ d_model = self.config.d_model+ B = src.shape[0]- # Full forward (no FFN slicing)encoder_out = self.encode_text(src, src_mask=src_mask)x_f32 = self._run_decoder(encoder_out, tgt, tgt_mask=tgt_mask)logits = x_f32 @ emb.T@@ -401,31 +406,37 @@ class EncoderDecoderTransformer(nn.Module):mrl_logits = []if mrl_dims is not None:for d in mrl_dims:- if d < self.config.d_model:- d_ff_slice = self.config.d_ff * d // self.config.d_model- enc_m = self.encode_text(src, src_mask=src_mask, d_ff_slice=d_ff_slice)- x_m = self._run_decoder(enc_m, tgt, tgt_mask=tgt_mask, d_ff_slice=d_ff_slice)+ if d < d_model:+ k = d_ff * d // d_model+ mask = (jnp.arange(d_ff) < k).astype(x_f32.dtype)+ mask = jnp.broadcast_to(mask[None, :], (B, d_ff))+ enc_m = self.encode_text(src, src_mask=src_mask, ffn_mask=mask)+ x_m = self._run_decoder(enc_m, tgt, tgt_mask=tgt_mask, ffn_mask=mask)mrl_logits.append(x_m @ emb.T)return logits, slot_div, mrl_logitsdef forward_speech_with_aux(self, mel, tgt, src_mask=None, tgt_mask=None, mrl_dims=None):+ """Eval-only: separate per-width speech forwards for reporting per-width PPL."""emb = self.embedding.embedding+ d_ff = self.config.d_ff+ d_model = self.config.d_model+ B = mel.shape[0]- # Full forward (no FFN slicing)encoder_out = self.encode_speech(mel, src_mask=src_mask)x_f32 = self._run_decoder(encoder_out, tgt, tgt_mask=tgt_mask)logits = x_f32 @ emb.Tslot_div = self._slot_diversity(encoder_out)- # MRL sub-model forwards with sliced FFNmrl_logits = []if mrl_dims is not None:for d in mrl_dims:- if d < self.config.d_model:- d_ff_slice = self.config.d_ff * d // self.config.d_model- enc_m = self.encode_speech(mel, src_mask=src_mask, d_ff_slice=d_ff_slice)- x_m = self._run_decoder(enc_m, tgt, tgt_mask=tgt_mask, d_ff_slice=d_ff_slice)+ if d < d_model:+ k = d_ff * d // d_model+ mask = (jnp.arange(d_ff) < k).astype(x_f32.dtype)+ mask = jnp.broadcast_to(mask[None, :], (B, d_ff))+ enc_m = self.encode_speech(mel, src_mask=src_mask, ffn_mask=mask)+ x_m = self._run_decoder(enc_m, tgt, tgt_mask=tgt_mask, ffn_mask=mask)mrl_logits.append(x_m @ emb.T)return logits, slot_div, mrl_logitsdiff --git a/src/train.py b/src/train.pyindex e15821c..ad2809d 100644--- a/src/train.py+++ b/src/train.py@@ -225,72 +225,80 @@ def _quantize_params(params, group_size=32):_GROUP_SIZE = 32_MRL_DIMS = ()+_MRL_FF_SLICES = () # precomputed d_ff slices, set alongside _MRL_DIMS+_D_FF = 2048 # set in train()-def _compute_loss(logits, slot_div, mrl_logits, tgt_out):- """Shared CE + MRL + z-loss + slot-div computation."""+def _text_loss_fn(state, params, src, tgt_in, tgt_out, causal_mask, ffn_mask):pad_id = 0+ src_mask = make_padding_mask(src, pad_id)+ tgt_mask = causal_mask & make_padding_mask(tgt_in, pad_id)+ logits, slot_div = state.apply_fn(+ {"params": _quantize_params(params, group_size=_GROUP_SIZE)},+ src, tgt_in, src_mask=src_mask, tgt_mask=tgt_mask,+ ffn_mask=ffn_mask,+ method="forward_masked",+ )logits_f32 = logits.astype(jnp.float32)mask = (tgt_out != pad_id).astype(jnp.float32)ce_loss = jnp.sum(optax.softmax_cross_entropy_with_integer_labels(logits_f32, tgt_out) * mask) / jnp.maximum(jnp.sum(mask), 1.0)-- total_loss = ce_loss- num_scales = 1- for mrl_log in mrl_logits:- mrl_log_f32 = mrl_log.astype(jnp.float32)- aux_loss = jnp.sum(- optax.softmax_cross_entropy_with_integer_labels(mrl_log_f32, tgt_out) * mask- ) / jnp.maximum(jnp.sum(mask), 1.0)- total_loss = total_loss + aux_loss- num_scales = num_scales + 1- total_loss = total_loss / num_scales-z_loss = 1e-4 * jnp.mean(jax.nn.logsumexp(logits_f32, axis=-1) ** 2)div_loss = 1e-4 * slot_div- return total_loss + z_loss + div_loss---def _text_loss_fn(state, params, src, tgt_in, tgt_out, causal_mask):- pad_id = 0- src_mask = make_padding_mask(src, pad_id)- tgt_mask = causal_mask & make_padding_mask(tgt_in, pad_id)-- logits, slot_div, mrl_logits = state.apply_fn(- {"params": _quantize_params(params, group_size=_GROUP_SIZE)},- src,- tgt_in,- src_mask=src_mask,- tgt_mask=tgt_mask,- cross_mask=src_mask,- mrl_dims=_MRL_DIMS if _MRL_DIMS else None,- method="forward_with_aux",- )- return _compute_loss(logits, slot_div, mrl_logits, tgt_out)+ return ce_loss + z_loss + div_loss-def _speech_loss_fn(state, params, mel, tgt_in, tgt_out, causal_mask):+def _speech_loss_fn(state, params, mel, tgt_in, tgt_out, causal_mask, ffn_mask):pad_id = 0src_mask = make_mel_padding_mask(mel)tgt_mask = causal_mask & make_padding_mask(tgt_in, pad_id)-- logits, slot_div, mrl_logits = state.apply_fn(+ logits, slot_div = state.apply_fn({"params": _quantize_params(params, group_size=_GROUP_SIZE)},- mel,- tgt_in,- src_mask=src_mask,- tgt_mask=tgt_mask,- mrl_dims=_MRL_DIMS if _MRL_DIMS else None,- method="forward_speech_with_aux",+ mel, tgt_in, src_mask=src_mask, tgt_mask=tgt_mask,+ ffn_mask=ffn_mask,+ method="forward_speech_masked",)- return _compute_loss(logits, slot_div, mrl_logits, tgt_out)+ logits_f32 = logits.astype(jnp.float32)+ mask = (tgt_out != pad_id).astype(jnp.float32)+ ce_loss = jnp.sum(+ optax.softmax_cross_entropy_with_integer_labels(logits_f32, tgt_out) * mask+ ) / jnp.maximum(jnp.sum(mask), 1.0)+ z_loss = 1e-4 * jnp.mean(jax.nn.logsumexp(logits_f32, axis=-1) ** 2)+ div_loss = 1e-4 * slot_div+ return ce_loss + z_loss + div_loss-def _train_step_text(state, ema_params, src, tgt_in, tgt_out, causal_mask):+def _make_ffn_mask(batch_size, d_ff, mrl_ff_slices):+ """Build (batch, d_ff) prefix mask: batch split equally across widths.++ First quarter = full width (all ones), remaining quarters = MRL slices.+ Each unique input is repeated once per width.+ """+ n_widths = 1 + len(mrl_ff_slices) # full + MRL dims+ per_width = batch_size // n_widths+ arange = jnp.arange(d_ff)+ rows = [jnp.ones((per_width, d_ff), dtype=jnp.bfloat16)] # full width+ for k in mrl_ff_slices:+ rows.append((arange[None, :] < k).astype(jnp.bfloat16).repeat(per_width, axis=0))+ # Handle remainder (assign to full width)+ remainder = batch_size - per_width * n_widths+ if remainder > 0:+ rows.append(jnp.ones((remainder, d_ff), dtype=jnp.bfloat16))+ return jnp.concatenate(rows, axis=0)+++def _repeat_for_mrl(arr, n_widths):+ """Repeat first batch/n_widths items n_widths times to fill batch."""+ per_width = arr.shape[0] // n_widths+ unique = arr[:per_width]+ return jnp.tile(unique, (n_widths,) + (1,) * (arr.ndim - 1))+++def _train_step_text(state, ema_params, src, tgt_in, tgt_out, causal_mask, ffn_mask):ema_decay = 0.999loss, grads = jax.value_and_grad(- lambda p: _text_loss_fn(state, p, src, tgt_in, tgt_out, causal_mask)+ lambda p: _text_loss_fn(state, p, src, tgt_in, tgt_out, causal_mask, ffn_mask))(state.params)grads = jax.lax.pmean(grads, axis_name="batch")loss = jax.lax.pmean(loss, axis_name="batch")@@ -300,11 +308,11 @@ def _train_step_text(state, ema_params, src, tgt_in, tgt_out, causal_mask):return state, ema_params, loss, grad_norm-def _train_step_text_masked(state, ema_params, src, tgt_in, tgt_out, causal_mask, prune_mask):- """Text training step with fused mask application."""+def _train_step_text_masked(state, ema_params, src, tgt_in, tgt_out, causal_mask, prune_mask, ffn_mask):+ """Text training step with fused prune mask application."""ema_decay = 0.999loss, grads = jax.value_and_grad(- lambda p: _text_loss_fn(state, p, src, tgt_in, tgt_out, causal_mask)+ lambda p: _text_loss_fn(state, p, src, tgt_in, tgt_out, causal_mask, ffn_mask))(state.params)grads = jax.lax.pmean(grads, axis_name="batch")loss = jax.lax.pmean(loss, axis_name="batch")@@ -316,10 +324,10 @@ def _train_step_text_masked(state, ema_params, src, tgt_in, tgt_out, causal_maskreturn state, ema_params, loss, grad_norm-def _train_step_speech(state, ema_params, mel, tgt_in, tgt_out, causal_mask):+def _train_step_speech(state, ema_params, mel, tgt_in, tgt_out, causal_mask, ffn_mask):ema_decay = 0.999loss, grads = jax.value_and_grad(- lambda p: _speech_loss_fn(state, p, mel, tgt_in, tgt_out, causal_mask)+ lambda p: _speech_loss_fn(state, p, mel, tgt_in, tgt_out, causal_mask, ffn_mask))(state.params)grads = jax.lax.pmean(grads, axis_name="batch")loss = jax.lax.pmean(loss, axis_name="batch")@@ -329,11 +337,11 @@ def _train_step_speech(state, ema_params, mel, tgt_in, tgt_out, causal_mask):return state, ema_params, loss, grad_norm-def _train_step_speech_masked(state, ema_params, mel, tgt_in, tgt_out, causal_mask, prune_mask):- """Speech training step with fused mask application."""+def _train_step_speech_masked(state, ema_params, mel, tgt_in, tgt_out, causal_mask, prune_mask, ffn_mask):+ """Speech training step with fused prune mask application."""ema_decay = 0.999loss, grads = jax.value_and_grad(- lambda p: _speech_loss_fn(state, p, mel, tgt_in, tgt_out, causal_mask)+ lambda p: _speech_loss_fn(state, p, mel, tgt_in, tgt_out, causal_mask, ffn_mask))(state.params)grads = jax.lax.pmean(grads, axis_name="batch")loss = jax.lax.pmean(loss, axis_name="batch")@@ -543,13 +551,17 @@ def train(args):n_mels=n_mels,)- global _GROUP_SIZE, _MRL_DIMS+ global _GROUP_SIZE, _MRL_DIMS, _MRL_FF_SLICES, _D_FF_GROUP_SIZE = getattr(args, "group_size", 32)+ _D_FF = config.d_ffmrl_dims_raw = getattr(args, "mrl_dims", None)if mrl_dims_raw:_MRL_DIMS = tuple(d for d in mrl_dims_raw if d < config.d_model)+ _MRL_FF_SLICES = tuple(config.d_ff * d // config.d_model for d in _MRL_DIMS)else:_MRL_DIMS = ()+ _MRL_FF_SLICES = ()+ n_widths = 1 + len(_MRL_FF_SLICES) if _MRL_FF_SLICES else 1p_train_step = _make_p_train_step()p_train_step_masked = _make_p_train_step_masked()p_train_step_speech = _make_p_train_step_speech()@@ -559,7 +571,11 @@ def train(args):rng = jax.random.PRNGKey(args.seed)rng, init_rng = jax.random.split(rng)- text_batches_per_epoch = len(enc_inputs) // effective_batch_size+ mrl_shared_input = getattr(args, "mrl_shared_input", False)+ # With shared input, each unique sample is repeated n_widths times,+ # so we fetch smaller batches but take more steps per epoch.+ unique_batch_size = effective_batch_size // n_widths if (mrl_shared_input and n_widths > 1) else effective_batch_size+ text_batches_per_epoch = len(enc_inputs) // unique_batch_sizespeech_batches_per_epoch = text_batches_per_epoch // speech_every if (not no_speech and speech_mel is not None) else 0num_batches = text_batches_per_epoch + speech_batches_per_epochtotal_steps = num_batches * args.epochs@@ -614,6 +630,20 @@ def train(args):(num_devices, 1, args.max_dec_len, args.max_dec_len),)+ # Precompute FFN mask for heterogeneous MRL training.+ # Batch is split: 1/N full-width + 1/N per MRL dim.+ text_ffn_mask = _make_ffn_mask(args.batch_size, config.d_ff, _MRL_FF_SLICES)+ text_ffn_mask = jnp.broadcast_to(+ text_ffn_mask[None, :, :],+ (num_devices, args.batch_size, config.d_ff),+ )+ if n_widths > 1:+ print(f" MRL widths {n_widths} (full + {', '.join(str(s) for s in _MRL_FF_SLICES)})")+ if mrl_shared_input:+ print(f" MRL mode shared input ({unique_batch_size // num_devices}/dev x {n_widths} repeats)")+ else:+ print(f" MRL mode unique input ({args.batch_size}/dev, random width)")+adam_schedule = _wsd_schedule(scaled_lr, total_steps, warmup_steps)muon_schedule = _wsd_schedule(muon_lr, total_steps, warmup_steps)tokens_per_batch = effective_batch_size * (args.max_enc_len + args.max_dec_len)@@ -643,13 +673,13 @@ def train(args):text_losses = []speech_losses = []- text_batches = get_batches(enc_inputs, dec_inputs, dec_targets, effective_batch_size)+ text_batches = get_batches(enc_inputs, dec_inputs, dec_targets, unique_batch_size)# Speech batches cycle via itertools.cyclespeech_iter = Noneif not no_speech and speech_mel is not None:speech_iter = iter(itertools.cycle(- get_speech_batches(speech_mel, speech_dec_in, speech_dec_tgt, effective_batch_size)+ get_speech_batches(speech_mel, speech_dec_in, speech_dec_tgt, unique_batch_size)))text_step_in_epoch = 0@@ -658,17 +688,25 @@ def train(args):for src, tgt_in, tgt_out in pbar:# --- Text step ---t0 = time.perf_counter()++ if n_widths > 1 and mrl_shared_input:+ # Shared: repeat each unique input across all widths+ src = np.tile(src, (n_widths, 1))+ tgt_in = np.tile(tgt_in, (n_widths, 1))+ tgt_out = np.tile(tgt_out, (n_widths, 1))+ # else: unique input per batch item, random width via ffn_mask+src_b = shard_batch(src, num_devices)tgt_in_b = shard_batch(tgt_in, num_devices)tgt_out_b = shard_batch(tgt_out, num_devices)if prune_mask is not None:state, ema_params, loss, grad_norm = p_train_step_masked(- state, ema_params, src_b, tgt_in_b, tgt_out_b, causal_mask, prune_mask,+ state, ema_params, src_b, tgt_in_b, tgt_out_b, causal_mask, prune_mask, text_ffn_mask,)else:state, ema_params, loss, grad_norm = p_train_step(- state, ema_params, src_b, tgt_in_b, tgt_out_b, causal_mask,+ state, ema_params, src_b, tgt_in_b, tgt_out_b, causal_mask, text_ffn_mask,)text_loss_val = float(loss[0])@@ -680,17 +718,26 @@ def train(args):speech_loss_val = Noneif speech_iter is not None and text_step_in_epoch % speech_every == 0:mel_batch, sp_tgt_in, sp_tgt_out = next(speech_iter)++ if n_widths > 1 and mrl_shared_input:+ mel_batch = np.tile(mel_batch, (n_widths, 1, 1))+ sp_tgt_in = np.tile(sp_tgt_in, (n_widths, 1))+ sp_tgt_out = np.tile(sp_tgt_out, (n_widths, 1))+mel_b = shard_batch(mel_batch, num_devices)sp_tgt_in_b = shard_batch(sp_tgt_in, num_devices)sp_tgt_out_b = shard_batch(sp_tgt_out, num_devices)+ # Speech ffn_mask: same width structure, but mel has different seq dim+ speech_ffn_mask = text_ffn_mask # same (devices, batch, d_ff) mask+if prune_mask is not None:state, ema_params, sp_loss, sp_grad_norm = p_train_step_speech_masked(- state, ema_params, mel_b, sp_tgt_in_b, sp_tgt_out_b, causal_mask, prune_mask,+ state, ema_params, mel_b, sp_tgt_in_b, sp_tgt_out_b, causal_mask, prune_mask, speech_ffn_mask,)else:state, ema_params, sp_loss, sp_grad_norm = p_train_step_speech(- state, ema_params, mel_b, sp_tgt_in_b, sp_tgt_out_b, causal_mask,+ state, ema_params, mel_b, sp_tgt_in_b, sp_tgt_out_b, causal_mask, speech_ffn_mask,)speech_loss_val = float(sp_loss[0])speech_losses.append(speech_loss_val)