Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

proper matryoshka and results

Noah Cylichcbc38056e69f
diff --git a/docs/matryoshka_results.md b/docs/matryoshka_results.md
new file mode 100644
index 0000000..8ee8627
--- /dev/null
+++ b/docs/matryoshka_results.md
@@ -0,0 +1,103 @@
+# FFN Interior Matryoshka — Results
+
+## Approach
+
+Heterogeneous batch MRL: a per-batch-item prefix mask zeros out FFN neurons beyond the target width. Single forward-backward pass, single optimizer step. All MRL widths trained simultaneously.
+
+- **Full width**: d_ff = 2048 (all neurons active)
+- **MRL 256**: d_ff = 1024 (first 1024 neurons)
+- **MRL 128**: d_ff = 512 (first 512 neurons)
+- **MRL 64**: d_ff = 256 (first 256 neurons)
+
+d_model, attention heads, embeddings, and mixer are unchanged across all widths.
+
+Two modes:
+- **Unique input** (default): each batch item is a unique sample, assigned a fixed width based on its position in the batch. Full data diversity per epoch.
+- **Shared input** (`--mrl-shared-input`): each unique sample is repeated across all widths. Smaller unique batch but every sample trains every width. Epoch = 4x more steps to see all data.
+
+## Training Config
+
+All runs use the same architecture and hyperparameters:
+
+| Setting | Value |
+|---|---|
+| Parameters | 87,666,176 |
+| d_model | 512 |
+| d_ff | 2048 |
+| Heads (KV) | 8 (8) |
+| Layers | 8 enc / 4 dec |
+| Memory slots | 64 |
+| Activation | dReLU |
+| Dtype | bfloat16 |
+| Devices | 8 (TPU v6e-8) |
+| Batch | 32 x 8 = 256 |
+| LR (Adam) | 0.0003 x 8 = 0.0024 |
+| LR (Muon) | 0.02 -> 0.0566 |
+| Sparsity | 50% target (gradual cubic, group=32) |
+| Speech | every 3 text steps |
+
+## Results (1 epoch, val PPL)
+
+| Model | Main (output-only slice) | FFN unique input | FFN shared input |
+|---|---|---|---|
+| Full (d_ff=2048) | **4.52** | 4.66 | 5.11 |
+| MRL 256 (d_ff=1024) | - | **4.66** | 5.11 |
+| MRL 128 (d_ff=512) | - | **4.68** | 5.11 |
+| MRL 64 (d_ff=256) | - | **4.80** | 5.20 |
+| Quant (INT4 g32) | **4.59** | 4.75 | 5.20 |
+
+> Main branch MRL PPLs omitted — output-only slice MRL does not constrain internal computation, so sub-model PPLs are misleading.
+
+| Metric | Main | FFN unique | FFN shared |
+|---|---|---|---|
+| MRL method | Output-only slice | FFN interior mask | FFN interior mask |
+| Total steps | 11,038 | 11,038 | 44,156 |
+| Unique items/batch | 32/dev | 32/dev | 8/dev x 4 |
+| Speed | ~19.3 it/s | 11.8 it/s | 20.9 it/s |
+| Wall time | ~10 min | ~16 min | ~36 min |
+| Avg train loss | 4.43 | 4.42 | 3.12 |
+| Final train loss | 1.51 | 1.53 | 1.66 |
+| Text val PPL | **4.52** | 4.66 | 5.11 |
+| Quant val PPL | **4.59** | 4.75 | 5.20 |
+| Weight sparsity | 50.02% | 50.01% | 68.28% |
+| Speech val PPL | 87.26 | 92.59 | 78.64 |
+| Speech WER | 1.0000 | 0.9695 | 0.9618 |
+| Throughput | 135.7 tok/s | 46.3 tok/s | 156.8 tok/s |
+
+## Analysis
+
+**Main branch full-model PPL (4.52) is the best**, as expected — it doesn't spend any capacity on sub-model training. FFN unique input (4.66) pays a +0.14 PPL cost for genuinely training 4 widths simultaneously.
+
+**FFN unique input beats shared input** (4.66 vs 5.11). Both see the same total unique data per epoch, but unique input has 4x data diversity per step (32 unique items vs 8 repeated 4x).
+
+**Shared input over-prunes** (68% vs 50% target) because the pruning schedule fractions apply to 4x more total steps, creating a wider absolute pruning window.
+
+**FFN interior MRL sub-models are tight.** With unique input, the d_ff=256 sub-model (1/8 of FFN, 68.7M params) is only +0.14 PPL worse than full. This is a genuine sub-network — unlike main's output-only approach which shares all internal computation.
+
+**Main is faster** (~19 it/s) because it runs one full forward + cheap logit slicing. FFN unique input (11.8 it/s) does one forward with heterogeneous masking but can't exploit XLA batch optimizations as well. FFN shared input (20.9 it/s) is fastest per-step due to repeated inputs.
+
+### Recommendation
+
+Use **unique input** (default, no flag) for best quality. The +0.14 PPL cost over main buys genuinely independent sub-models that can be exported at reduced FFN width.
+
+## Training Speed
+
+| Approach | it/s | Notes |
+|---|---|---|
+| Main branch (output-only slice) | ~19.3 | 1 forward + logit slicing (no true sub-models) |
+| FFN unique input | 11.8 | Single forward, heterogeneous batch |
+| FFN shared input | 20.9 | Single forward, repeated inputs (XLA optimizes) |
+
+## Generation Samples (Unique Input)
+
+```
+<story>[Once upon a time] where gan the vessel was calculated with a sensible light
+<story>[The little dog] and the big boots and buntline furnished with a very little red buttons and si
+<story>[She was very happy because] but where if you ever see her again i know she has a very special thing i'm go
+```
+
+## Notes
+
+- All runs are 1-epoch on TinyStories + LibriSpeech. Multi-epoch runs would improve all metrics.
+- Speech WER is poor across all methods — expected with interleaved training and limited speech data.
+- The heterogeneous batch approach uses the same memory as a single full-width forward (~1x). No overhead from MRL.
diff --git a/src/cli.py b/src/cli.py
index e912af0..93bef0c 100644
--- a/src/cli.py
+++ b/src/cli.py
@@ -120,6 +120,8 @@ def main():
p.add_argument("--num-memory-slots", type=int, default=64)
p.add_argument("--mrl-dims", type=int, nargs="*", default=[256, 128, 64],
help="MRL dimension targets (default: 256 128 64)")
+ p.add_argument("--mrl-shared-input", action="store_true",
+ help="Each unique input is repeated across all MRL widths (default: unique input per width)")
p.add_argument("--no-speech", action="store_true", help="Disable speech training (text-only)")
p.add_argument("--speech-every", type=int, default=3,
help="Do one speech step every N text steps (default: 3)")
diff --git a/src/model.py b/src/model.py
index 38b0b2a..d8c27e9 100644
--- a/src/model.py
+++ b/src/model.py
@@ -129,31 +129,17 @@ class FeedForward(nn.Module):
activation: str = "drelu"
@nn.compact
- def __call__(self, x, d_ff_slice=None):
- sliced = d_ff_slice is not None and d_ff_slice < self.d_ff
-
- if sliced:
- # Sliced path: access existing kernels and slice for efficiency.
- # Params must already exist from a prior full-width call (e.g. the
- # non-MRL forward in forward_with_aux).
- gk = self.variables['params']['gate_proj']['kernel'][:, :d_ff_slice]
- uk = self.variables['params']['up_proj']['kernel'][:, :d_ff_slice]
- gate = x.astype(self.dtype) @ gk.astype(self.dtype)
- up = x.astype(self.dtype) @ uk.astype(self.dtype)
- else:
- gate = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="gate_proj")(x)
- up = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="up_proj")(x)
-
+ def __call__(self, x, ffn_mask=None):
+ gate = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="gate_proj")(x)
+ up = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="up_proj")(x)
if self.activation == "swiglu":
h = nn.silu(gate) * up
elif self.activation == "geglu":
h = nn.gelu(gate) * up
else: # drelu
h = nn.relu(gate) * nn.relu(up)
-
- if sliced:
- dk = self.variables['params']['down_proj']['kernel'][:d_ff_slice]
- return (h @ dk.astype(self.dtype))
+ if ffn_mask is not None:
+ h = h * ffn_mask[:, None, :] # (batch, 1, d_ff)
return nn.Dense(self.d_model, dtype=self.dtype, use_bias=False, kernel_init=residual_init(self.num_layers), name="down_proj")(h)
@@ -210,7 +196,7 @@ class MemoryMixerBlock(nn.Module):
activation: str = "drelu"
@nn.compact
- def __call__(self, x, s, mask=None, rope=None, d_ff_slice=None):
+ def __call__(self, x, s, mask=None, rope=None, ffn_mask=None):
s_norm = ZCRMSNorm(dtype=self.dtype, name="pack_s_norm")(s)
x_norm = ZCRMSNorm(dtype=self.dtype, name="pack_x_norm")(x)
s = s + MultiHeadAttention(
@@ -224,7 +210,7 @@ class MemoryMixerBlock(nn.Module):
residual = x
x = ZCRMSNorm(dtype=self.dtype, name="local_norm")(x)
- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation, name="local_ffn")(x, d_ff_slice=d_ff_slice)
+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation, name="local_ffn")(x, ffn_mask=ffn_mask)
x = x + residual
return x, s
@@ -235,7 +221,7 @@ class MemoryMixerEncoder(nn.Module):
config: TransformerConfig
@nn.compact
- def __call__(self, x, mask=None, rope=None, d_ff_slice=None):
+ def __call__(self, x, mask=None, rope=None, ffn_mask=None):
cfg = self.config
dt = cfg.jax_dtype
x = x.astype(dt)
@@ -262,7 +248,7 @@ class MemoryMixerEncoder(nn.Module):
x, s = nn.remat(MemoryMixerBlock)(
cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff,
cfg.num_memory_slots, cfg.total_layers, dt, cfg.activation, name=f"block_{i}"
- )(x, s, mask=mask, rope=rope, d_ff_slice=d_ff_slice)
+ )(x, s, mask=mask, rope=rope, ffn_mask=ffn_mask)
s = ZCRMSNorm(dtype=dt, name="final_norm")(s)
return s
@@ -278,7 +264,7 @@ class DecoderBlock(nn.Module):
activation: str = "drelu"
@nn.compact
- def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, d_ff_slice=None):
+ def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, ffn_mask=None):
residual = x
x = ZCRMSNorm(dtype=self.dtype)(x)
x = MultiHeadAttention(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers, self.dtype, name="self_attn")(
@@ -295,7 +281,7 @@ class DecoderBlock(nn.Module):
residual = x
x = ZCRMSNorm(dtype=self.dtype)(x)
- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, d_ff_slice=d_ff_slice)
+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)
x = x + residual
return x
@@ -306,7 +292,7 @@ class Decoder(nn.Module):
config: TransformerConfig
@nn.compact
- def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, d_ff_slice=None):
+ def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, ffn_mask=None):
cfg = self.config
dt = cfg.jax_dtype
x = x.astype(dt)
@@ -314,7 +300,7 @@ class Decoder(nn.Module):
for i in range(cfg.num_decoder_layers):
x = nn.remat(DecoderBlock)(
cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, dt, cfg.activation, name=f"block_{i}"
- )(x, encoder_out, self_mask=self_mask, cross_mask=cross_mask, rope=rope, d_ff_slice=d_ff_slice)
+ )(x, encoder_out, self_mask=self_mask, cross_mask=cross_mask, rope=rope, ffn_mask=ffn_mask)
x = ZCRMSNorm(dtype=dt)(x)
return x
@@ -346,19 +332,19 @@ class EncoderDecoderTransformer(nn.Module):
head_dim = self.config.d_model // self.config.num_heads
return precompute_rope_freqs(head_dim, seq_len, self.config.rope_theta)
- def encode_text(self, src, src_mask=None, d_ff_slice=None):
+ def encode_text(self, src, src_mask=None, ffn_mask=None):
x = self.embedding(src) * self.embed_scale
rope = self._rope(src.shape[1])
- return self.encoder(x, mask=src_mask, rope=rope, d_ff_slice=d_ff_slice)
+ return self.encoder(x, mask=src_mask, rope=rope, ffn_mask=ffn_mask)
def encode(self, src, src_mask=None):
"""Backward-compatible alias for encode_text."""
return self.encode_text(src, src_mask=src_mask)
- def encode_speech(self, mel, src_mask=None, d_ff_slice=None):
+ def encode_speech(self, mel, src_mask=None, ffn_mask=None):
x = self.mel_proj(mel) * self.embed_scale
rope = self._rope(mel.shape[1])
- return self.encoder(x, mask=src_mask, rope=rope, d_ff_slice=d_ff_slice)
+ return self.encoder(x, mask=src_mask, rope=rope, ffn_mask=ffn_mask)
def decode(self, tgt, encoder_out, self_mask=None, cross_mask=None):
x = self.embedding(tgt) * self.embed_scale
@@ -376,11 +362,11 @@ class EncoderDecoderTransformer(nn.Module):
)
return logits
- def _run_decoder(self, encoder_out, tgt, tgt_mask=None, d_ff_slice=None):
+ def _run_decoder(self, encoder_out, tgt, tgt_mask=None, ffn_mask=None):
"""Run decoder and return float32 hidden states."""
x = self.embedding(tgt) * self.embed_scale
rope = self._rope(tgt.shape[1])
- x = self.decoder(x, encoder_out, self_mask=tgt_mask, cross_mask=None, rope=rope, d_ff_slice=d_ff_slice)
+ x = self.decoder(x, encoder_out, self_mask=tgt_mask, cross_mask=None, rope=rope, ffn_mask=ffn_mask)
return x.astype(jnp.float32)
def _slot_diversity(self, encoder_out):
@@ -389,10 +375,29 @@ class EncoderDecoderTransformer(nn.Module):
diag_sq = jnp.sum(jnp.diagonal(gram, axis1=1, axis2=2) ** 2)
return (jnp.sum(gram ** 2) - diag_sq) / s.shape[0]
+ def forward_masked(self, src, tgt, src_mask=None, tgt_mask=None, ffn_mask=None):
+ """Single forward with per-batch-item FFN masking. Returns (logits, slot_div)."""
+ encoder_out = self.encode_text(src, src_mask=src_mask, ffn_mask=ffn_mask)
+ x_f32 = self._run_decoder(encoder_out, tgt, tgt_mask=tgt_mask, ffn_mask=ffn_mask)
+ logits = x_f32 @ self.embedding.embedding.T
+ slot_div = self._slot_diversity(encoder_out)
+ return logits, slot_div
+
+ def forward_speech_masked(self, mel, tgt, src_mask=None, tgt_mask=None, ffn_mask=None):
+ """Single speech forward with per-batch-item FFN masking. Returns (logits, slot_div)."""
+ encoder_out = self.encode_speech(mel, src_mask=src_mask, ffn_mask=ffn_mask)
+ x_f32 = self._run_decoder(encoder_out, tgt, tgt_mask=tgt_mask, ffn_mask=ffn_mask)
+ logits = x_f32 @ self.embedding.embedding.T
+ slot_div = self._slot_diversity(encoder_out)
+ return logits, slot_div
+
def forward_with_aux(self, src, tgt, src_mask=None, tgt_mask=None, cross_mask=None, mrl_dims=None):
+ """Eval-only: separate per-width forwards for reporting per-width PPL."""
emb = self.embedding.embedding
+ d_ff = self.config.d_ff
+ d_model = self.config.d_model
+ B = src.shape[0]
- # Full forward (no FFN slicing)
encoder_out = self.encode_text(src, src_mask=src_mask)
x_f32 = self._run_decoder(encoder_out, tgt, tgt_mask=tgt_mask)
logits = x_f32 @ emb.T
@@ -401,31 +406,37 @@ class EncoderDecoderTransformer(nn.Module):
mrl_logits = []
if mrl_dims is not None:
for d in mrl_dims:
- if d < self.config.d_model:
- d_ff_slice = self.config.d_ff * d // self.config.d_model
- enc_m = self.encode_text(src, src_mask=src_mask, d_ff_slice=d_ff_slice)
- x_m = self._run_decoder(enc_m, tgt, tgt_mask=tgt_mask, d_ff_slice=d_ff_slice)
+ if d < d_model:
+ k = d_ff * d // d_model
+ mask = (jnp.arange(d_ff) < k).astype(x_f32.dtype)
+ mask = jnp.broadcast_to(mask[None, :], (B, d_ff))
+ enc_m = self.encode_text(src, src_mask=src_mask, ffn_mask=mask)
+ x_m = self._run_decoder(enc_m, tgt, tgt_mask=tgt_mask, ffn_mask=mask)
mrl_logits.append(x_m @ emb.T)
return logits, slot_div, mrl_logits
def forward_speech_with_aux(self, mel, tgt, src_mask=None, tgt_mask=None, mrl_dims=None):
+ """Eval-only: separate per-width speech forwards for reporting per-width PPL."""
emb = self.embedding.embedding
+ d_ff = self.config.d_ff
+ d_model = self.config.d_model
+ B = mel.shape[0]
- # Full forward (no FFN slicing)
encoder_out = self.encode_speech(mel, src_mask=src_mask)
x_f32 = self._run_decoder(encoder_out, tgt, tgt_mask=tgt_mask)
logits = x_f32 @ emb.T
slot_div = self._slot_diversity(encoder_out)
- # MRL sub-model forwards with sliced FFN
mrl_logits = []
if mrl_dims is not None:
for d in mrl_dims:
- if d < self.config.d_model:
- d_ff_slice = self.config.d_ff * d // self.config.d_model
- enc_m = self.encode_speech(mel, src_mask=src_mask, d_ff_slice=d_ff_slice)
- x_m = self._run_decoder(enc_m, tgt, tgt_mask=tgt_mask, d_ff_slice=d_ff_slice)
+ if d < d_model:
+ k = d_ff * d // d_model
+ mask = (jnp.arange(d_ff) < k).astype(x_f32.dtype)
+ mask = jnp.broadcast_to(mask[None, :], (B, d_ff))
+ enc_m = self.encode_speech(mel, src_mask=src_mask, ffn_mask=mask)
+ x_m = self._run_decoder(enc_m, tgt, tgt_mask=tgt_mask, ffn_mask=mask)
mrl_logits.append(x_m @ emb.T)
return logits, slot_div, mrl_logits
diff --git a/src/train.py b/src/train.py
index e15821c..ad2809d 100644
--- a/src/train.py
+++ b/src/train.py
@@ -225,72 +225,80 @@ def _quantize_params(params, group_size=32):
_GROUP_SIZE = 32
_MRL_DIMS = ()
+_MRL_FF_SLICES = () # precomputed d_ff slices, set alongside _MRL_DIMS
+_D_FF = 2048 # set in train()
-def _compute_loss(logits, slot_div, mrl_logits, tgt_out):
- """Shared CE + MRL + z-loss + slot-div computation."""
+def _text_loss_fn(state, params, src, tgt_in, tgt_out, causal_mask, ffn_mask):
pad_id = 0
+ src_mask = make_padding_mask(src, pad_id)
+ tgt_mask = causal_mask & make_padding_mask(tgt_in, pad_id)
+ logits, slot_div = state.apply_fn(
+ {"params": _quantize_params(params, group_size=_GROUP_SIZE)},
+ src, tgt_in, src_mask=src_mask, tgt_mask=tgt_mask,
+ ffn_mask=ffn_mask,
+ method="forward_masked",
+ )
logits_f32 = logits.astype(jnp.float32)
mask = (tgt_out != pad_id).astype(jnp.float32)
ce_loss = jnp.sum(
optax.softmax_cross_entropy_with_integer_labels(logits_f32, tgt_out) * mask
) / jnp.maximum(jnp.sum(mask), 1.0)
-
- total_loss = ce_loss
- num_scales = 1
- for mrl_log in mrl_logits:
- mrl_log_f32 = mrl_log.astype(jnp.float32)
- aux_loss = jnp.sum(
- optax.softmax_cross_entropy_with_integer_labels(mrl_log_f32, tgt_out) * mask
- ) / jnp.maximum(jnp.sum(mask), 1.0)
- total_loss = total_loss + aux_loss
- num_scales = num_scales + 1
- total_loss = total_loss / num_scales
-
z_loss = 1e-4 * jnp.mean(jax.nn.logsumexp(logits_f32, axis=-1) ** 2)
div_loss = 1e-4 * slot_div
- return total_loss + z_loss + div_loss
-
-
-def _text_loss_fn(state, params, src, tgt_in, tgt_out, causal_mask):
- pad_id = 0
- src_mask = make_padding_mask(src, pad_id)
- tgt_mask = causal_mask & make_padding_mask(tgt_in, pad_id)
-
- logits, slot_div, mrl_logits = state.apply_fn(
- {"params": _quantize_params(params, group_size=_GROUP_SIZE)},
- src,
- tgt_in,
- src_mask=src_mask,
- tgt_mask=tgt_mask,
- cross_mask=src_mask,
- mrl_dims=_MRL_DIMS if _MRL_DIMS else None,
- method="forward_with_aux",
- )
- return _compute_loss(logits, slot_div, mrl_logits, tgt_out)
+ return ce_loss + z_loss + div_loss
-def _speech_loss_fn(state, params, mel, tgt_in, tgt_out, causal_mask):
+def _speech_loss_fn(state, params, mel, tgt_in, tgt_out, causal_mask, ffn_mask):
pad_id = 0
src_mask = make_mel_padding_mask(mel)
tgt_mask = causal_mask & make_padding_mask(tgt_in, pad_id)
-
- logits, slot_div, mrl_logits = state.apply_fn(
+ logits, slot_div = state.apply_fn(
{"params": _quantize_params(params, group_size=_GROUP_SIZE)},
- mel,
- tgt_in,
- src_mask=src_mask,
- tgt_mask=tgt_mask,
- mrl_dims=_MRL_DIMS if _MRL_DIMS else None,
- method="forward_speech_with_aux",
+ mel, tgt_in, src_mask=src_mask, tgt_mask=tgt_mask,
+ ffn_mask=ffn_mask,
+ method="forward_speech_masked",
)
- return _compute_loss(logits, slot_div, mrl_logits, tgt_out)
+ logits_f32 = logits.astype(jnp.float32)
+ mask = (tgt_out != pad_id).astype(jnp.float32)
+ ce_loss = jnp.sum(
+ optax.softmax_cross_entropy_with_integer_labels(logits_f32, tgt_out) * mask
+ ) / jnp.maximum(jnp.sum(mask), 1.0)
+ z_loss = 1e-4 * jnp.mean(jax.nn.logsumexp(logits_f32, axis=-1) ** 2)
+ div_loss = 1e-4 * slot_div
+ return ce_loss + z_loss + div_loss
-def _train_step_text(state, ema_params, src, tgt_in, tgt_out, causal_mask):
+def _make_ffn_mask(batch_size, d_ff, mrl_ff_slices):
+ """Build (batch, d_ff) prefix mask: batch split equally across widths.
+
+ First quarter = full width (all ones), remaining quarters = MRL slices.
+ Each unique input is repeated once per width.
+ """
+ n_widths = 1 + len(mrl_ff_slices) # full + MRL dims
+ per_width = batch_size // n_widths
+ arange = jnp.arange(d_ff)
+ rows = [jnp.ones((per_width, d_ff), dtype=jnp.bfloat16)] # full width
+ for k in mrl_ff_slices:
+ rows.append((arange[None, :] < k).astype(jnp.bfloat16).repeat(per_width, axis=0))
+ # Handle remainder (assign to full width)
+ remainder = batch_size - per_width * n_widths
+ if remainder > 0:
+ rows.append(jnp.ones((remainder, d_ff), dtype=jnp.bfloat16))
+ return jnp.concatenate(rows, axis=0)
+
+
+def _repeat_for_mrl(arr, n_widths):
+ """Repeat first batch/n_widths items n_widths times to fill batch."""
+ per_width = arr.shape[0] // n_widths
+ unique = arr[:per_width]
+ return jnp.tile(unique, (n_widths,) + (1,) * (arr.ndim - 1))
+
+
+def _train_step_text(state, ema_params, src, tgt_in, tgt_out, causal_mask, ffn_mask):
ema_decay = 0.999
loss, grads = jax.value_and_grad(
- lambda p: _text_loss_fn(state, p, src, tgt_in, tgt_out, causal_mask)
+ lambda p: _text_loss_fn(state, p, src, tgt_in, tgt_out, causal_mask, ffn_mask)
)(state.params)
grads = jax.lax.pmean(grads, axis_name="batch")
loss = jax.lax.pmean(loss, axis_name="batch")
@@ -300,11 +308,11 @@ def _train_step_text(state, ema_params, src, tgt_in, tgt_out, causal_mask):
return state, ema_params, loss, grad_norm
-def _train_step_text_masked(state, ema_params, src, tgt_in, tgt_out, causal_mask, prune_mask):
- """Text training step with fused mask application."""
+def _train_step_text_masked(state, ema_params, src, tgt_in, tgt_out, causal_mask, prune_mask, ffn_mask):
+ """Text training step with fused prune mask application."""
ema_decay = 0.999
loss, grads = jax.value_and_grad(
- lambda p: _text_loss_fn(state, p, src, tgt_in, tgt_out, causal_mask)
+ lambda p: _text_loss_fn(state, p, src, tgt_in, tgt_out, causal_mask, ffn_mask)
)(state.params)
grads = jax.lax.pmean(grads, axis_name="batch")
loss = jax.lax.pmean(loss, axis_name="batch")
@@ -316,10 +324,10 @@ def _train_step_text_masked(state, ema_params, src, tgt_in, tgt_out, causal_mask
return state, ema_params, loss, grad_norm
-def _train_step_speech(state, ema_params, mel, tgt_in, tgt_out, causal_mask):
+def _train_step_speech(state, ema_params, mel, tgt_in, tgt_out, causal_mask, ffn_mask):
ema_decay = 0.999
loss, grads = jax.value_and_grad(
- lambda p: _speech_loss_fn(state, p, mel, tgt_in, tgt_out, causal_mask)
+ lambda p: _speech_loss_fn(state, p, mel, tgt_in, tgt_out, causal_mask, ffn_mask)
)(state.params)
grads = jax.lax.pmean(grads, axis_name="batch")
loss = jax.lax.pmean(loss, axis_name="batch")
@@ -329,11 +337,11 @@ def _train_step_speech(state, ema_params, mel, tgt_in, tgt_out, causal_mask):
return state, ema_params, loss, grad_norm
-def _train_step_speech_masked(state, ema_params, mel, tgt_in, tgt_out, causal_mask, prune_mask):
- """Speech training step with fused mask application."""
+def _train_step_speech_masked(state, ema_params, mel, tgt_in, tgt_out, causal_mask, prune_mask, ffn_mask):
+ """Speech training step with fused prune mask application."""
ema_decay = 0.999
loss, grads = jax.value_and_grad(
- lambda p: _speech_loss_fn(state, p, mel, tgt_in, tgt_out, causal_mask)
+ lambda p: _speech_loss_fn(state, p, mel, tgt_in, tgt_out, causal_mask, ffn_mask)
)(state.params)
grads = jax.lax.pmean(grads, axis_name="batch")
loss = jax.lax.pmean(loss, axis_name="batch")
@@ -543,13 +551,17 @@ def train(args):
n_mels=n_mels,
)
- global _GROUP_SIZE, _MRL_DIMS
+ global _GROUP_SIZE, _MRL_DIMS, _MRL_FF_SLICES, _D_FF
_GROUP_SIZE = getattr(args, "group_size", 32)
+ _D_FF = config.d_ff
mrl_dims_raw = getattr(args, "mrl_dims", None)
if mrl_dims_raw:
_MRL_DIMS = tuple(d for d in mrl_dims_raw if d < config.d_model)
+ _MRL_FF_SLICES = tuple(config.d_ff * d // config.d_model for d in _MRL_DIMS)
else:
_MRL_DIMS = ()
+ _MRL_FF_SLICES = ()
+ n_widths = 1 + len(_MRL_FF_SLICES) if _MRL_FF_SLICES else 1
p_train_step = _make_p_train_step()
p_train_step_masked = _make_p_train_step_masked()
p_train_step_speech = _make_p_train_step_speech()
@@ -559,7 +571,11 @@ def train(args):
rng = jax.random.PRNGKey(args.seed)
rng, init_rng = jax.random.split(rng)
- text_batches_per_epoch = len(enc_inputs) // effective_batch_size
+ mrl_shared_input = getattr(args, "mrl_shared_input", False)
+ # With shared input, each unique sample is repeated n_widths times,
+ # so we fetch smaller batches but take more steps per epoch.
+ unique_batch_size = effective_batch_size // n_widths if (mrl_shared_input and n_widths > 1) else effective_batch_size
+ text_batches_per_epoch = len(enc_inputs) // unique_batch_size
speech_batches_per_epoch = text_batches_per_epoch // speech_every if (not no_speech and speech_mel is not None) else 0
num_batches = text_batches_per_epoch + speech_batches_per_epoch
total_steps = num_batches * args.epochs
@@ -614,6 +630,20 @@ def train(args):
(num_devices, 1, args.max_dec_len, args.max_dec_len),
)
+ # Precompute FFN mask for heterogeneous MRL training.
+ # Batch is split: 1/N full-width + 1/N per MRL dim.
+ text_ffn_mask = _make_ffn_mask(args.batch_size, config.d_ff, _MRL_FF_SLICES)
+ text_ffn_mask = jnp.broadcast_to(
+ text_ffn_mask[None, :, :],
+ (num_devices, args.batch_size, config.d_ff),
+ )
+ if n_widths > 1:
+ print(f" MRL widths {n_widths} (full + {', '.join(str(s) for s in _MRL_FF_SLICES)})")
+ if mrl_shared_input:
+ print(f" MRL mode shared input ({unique_batch_size // num_devices}/dev x {n_widths} repeats)")
+ else:
+ print(f" MRL mode unique input ({args.batch_size}/dev, random width)")
+
adam_schedule = _wsd_schedule(scaled_lr, total_steps, warmup_steps)
muon_schedule = _wsd_schedule(muon_lr, total_steps, warmup_steps)
tokens_per_batch = effective_batch_size * (args.max_enc_len + args.max_dec_len)
@@ -643,13 +673,13 @@ def train(args):
text_losses = []
speech_losses = []
- text_batches = get_batches(enc_inputs, dec_inputs, dec_targets, effective_batch_size)
+ text_batches = get_batches(enc_inputs, dec_inputs, dec_targets, unique_batch_size)
# Speech batches cycle via itertools.cycle
speech_iter = None
if not no_speech and speech_mel is not None:
speech_iter = iter(itertools.cycle(
- get_speech_batches(speech_mel, speech_dec_in, speech_dec_tgt, effective_batch_size)
+ get_speech_batches(speech_mel, speech_dec_in, speech_dec_tgt, unique_batch_size)
))
text_step_in_epoch = 0
@@ -658,17 +688,25 @@ def train(args):
for src, tgt_in, tgt_out in pbar:
# --- Text step ---
t0 = time.perf_counter()
+
+ if n_widths > 1 and mrl_shared_input:
+ # Shared: repeat each unique input across all widths
+ src = np.tile(src, (n_widths, 1))
+ tgt_in = np.tile(tgt_in, (n_widths, 1))
+ tgt_out = np.tile(tgt_out, (n_widths, 1))
+ # else: unique input per batch item, random width via ffn_mask
+
src_b = shard_batch(src, num_devices)
tgt_in_b = shard_batch(tgt_in, num_devices)
tgt_out_b = shard_batch(tgt_out, num_devices)
if prune_mask is not None:
state, ema_params, loss, grad_norm = p_train_step_masked(
- state, ema_params, src_b, tgt_in_b, tgt_out_b, causal_mask, prune_mask,
+ state, ema_params, src_b, tgt_in_b, tgt_out_b, causal_mask, prune_mask, text_ffn_mask,
)
else:
state, ema_params, loss, grad_norm = p_train_step(
- state, ema_params, src_b, tgt_in_b, tgt_out_b, causal_mask,
+ state, ema_params, src_b, tgt_in_b, tgt_out_b, causal_mask, text_ffn_mask,
)
text_loss_val = float(loss[0])
@@ -680,17 +718,26 @@ def train(args):
speech_loss_val = None
if speech_iter is not None and text_step_in_epoch % speech_every == 0:
mel_batch, sp_tgt_in, sp_tgt_out = next(speech_iter)
+
+ if n_widths > 1 and mrl_shared_input:
+ mel_batch = np.tile(mel_batch, (n_widths, 1, 1))
+ sp_tgt_in = np.tile(sp_tgt_in, (n_widths, 1))
+ sp_tgt_out = np.tile(sp_tgt_out, (n_widths, 1))
+
mel_b = shard_batch(mel_batch, num_devices)
sp_tgt_in_b = shard_batch(sp_tgt_in, num_devices)
sp_tgt_out_b = shard_batch(sp_tgt_out, num_devices)
+ # Speech ffn_mask: same width structure, but mel has different seq dim
+ speech_ffn_mask = text_ffn_mask # same (devices, batch, d_ff) mask
+
if prune_mask is not None:
state, ema_params, sp_loss, sp_grad_norm = p_train_step_speech_masked(
- state, ema_params, mel_b, sp_tgt_in_b, sp_tgt_out_b, causal_mask, prune_mask,
+ state, ema_params, mel_b, sp_tgt_in_b, sp_tgt_out_b, causal_mask, prune_mask, speech_ffn_mask,
)
else:
state, ema_params, sp_loss, sp_grad_norm = p_train_step_speech(
- state, ema_params, mel_b, sp_tgt_in_b, sp_tgt_out_b, causal_mask,
+ state, ema_params, mel_b, sp_tgt_in_b, sp_tgt_out_b, causal_mask, speech_ffn_mask,
)
speech_loss_val = float(sp_loss[0])
speech_losses.append(speech_loss_val)