Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
Sparsification
HenryNdubuaku7ccad3128e45
diff --git a/experiements/pruning.txt b/experiements/pruning.txtindex 2a5f8e9..a3ca264 100644--- a/experiements/pruning.txt+++ b/experiements/pruning.txt@@ -19,8 +19,9 @@ Independent variables:B. Layer prune ratio l ∈ {0.0, 0.125, 0.25, 0.375, 0.50}C. Group size g ∈ {16, 32, 64, 128}D. Pruning schedule ∈ {staged, simultaneous, reverse-order}- E. Model scale ∈ {BASE-256, LARGE-512}- F. QAT ∈ {off, int4-group32}+ E. FFN activation ∈ {dReLU, SwiGLU, GeGLU}+ F. Model scale ∈ {BASE-256, LARGE-512}+ G. QAT ∈ {off, int4-group32}Dependent variables (measured):- Validation perplexity (TinyStories)@@ -263,7 +264,65 @@ Deliverables:bits and 32/g is amortized scale overhead per weight.--------------------------------------------------------------------------------GROUP 9: Inference Efficiency Validation (Table 8, Figure 7)+GROUP 9: Activation Function × Pruning Interaction (Table 8, Figure 8)+ *** KEY FINDING ***+-------------------------------------------------------------------------------+Purpose: Test whether activation function choice determines pruning robustness.+Early observation: dReLU dramatically increases post-prune perplexity compared+to SwiGLU. This group systematically characterises the interaction.++Activation definitions:+ - dReLU: ReLU(gate) * ReLU(up) — naturally ~75% sparse activations+ - SwiGLU: SiLU(gate) * up — smooth, dense activations+ - GeGLU: GELU(gate) * up — smooth, nearly dense activations++Requires: --activation CLI flag wired to FeedForward module.++ 9a. Dense baselines (no pruning) — isolate activation quality+ For act in {drelu, swiglu, geglu}:+ needle train --epochs 3 --activation {act} --sparsity-ratio 0.0 \+ --layer-prune-ratio 0.0 --seed {42,137,256}++ 9b. Sparsification only — test pruning robustness per activation+ For act in {drelu, swiglu, geglu}, s in {0.25, 0.50}:+ needle train --epochs 3 --activation {act} --sparsity-ratio {s} \+ --seed {42,137,256}++ 9c. Joint sparsification + layer pruning — full compression+ For act in {drelu, swiglu, geglu}:+ needle train --epochs 4 --activation {act} --sparsity-ratio 0.25 \+ --layer-prune-ratio 0.25 --seed {42,137,256}++ 9d. LARGE-512 validation on best and worst activation from 9b-9c+ needle train --epochs 4 --d-model 512 --num-heads 8 --num-layers 8 \+ --activation {act} --sparsity-ratio 0.25 --layer-prune-ratio 0.25 \+ --seed {42,137,256}++Deliverables:+ - Grouped bar chart: activation × {dense, s=0.25, s=0.50} → val ppl (Fig 8a)+ - Table: full metrics for all activation × compression combos+ - Relative degradation plot: (pruned_ppl / dense_ppl) per activation (Fig 8b)+ — THIS IS THE KEY CHART: shows pruning *robustness* not just raw quality+ - Analysis of competing hypotheses:+ (a) dReLU hurts pruning because its ~75% activation sparsity means fewer+ active paths compensate for zeroed weights — the network has less+ redundancy to absorb pruning damage+ (b) dReLU helps pruning because its natural sparsity pattern means weight+ blocks that process mostly-zero activations are truly unimportant+ and can be safely pruned (better signal for the pruning criterion)+ (c) Smooth activations (SwiGLU/GeGLU) spread information across more+ weights, creating redundancy that makes the model more robust to+ weight removal but harder to compress efficiently+ - Recommendation: which activation to use when pruning is planned vs not++ Additional metric for this group:+ - Activation sparsity: measure % of zeros in FFN intermediate activations+ for each activation function at each pruning level. Report as:+ act_sparsity = E[fraction of zeros in ReLU(gate) * ReLU(up)]+ This connects weight sparsity to activation sparsity.++-------------------------------------------------------------------------------+GROUP 10: Inference Efficiency Validation (Table 9, Figure 9)-------------------------------------------------------------------------------Purpose: Translate theoretical compression into real speedup.@@ -315,7 +374,8 @@ Tables:T5. Schedule ablation (3 schedules)T6. QAT interaction (4 methods × {on,off})T7. Group size ablation (4 sizes × 2 sparsity levels + joint)- T8. Inference efficiency (Pareto-optimal checkpoints)+ T8. Activation × pruning interaction (3 activations × 4 compression levels)+ T9. Inference efficiency (Pareto-optimal checkpoints)Figures:F1. Sparsity-only Pareto: sparsity % vs perplexity + LAMBADA@@ -325,6 +385,8 @@ Figures:F5. Compression ratio vs actual inference speedupF6. Group size: (a) g vs perplexity at s=0.25/0.50, (b) g vs LAMBADAF7. Effective bits-per-param vs quality across group sizes+ F8. Activation × pruning: (a) grouped bars, (b) relative degradation ratio+ F9. Activation sparsity vs weight sparsity interaction================================================================================SECTION 6 — COMPUTE BUDGET@@ -342,11 +404,13 @@ Estimated total runs:Group 6: 5 epoch counts × 3 seeds = 15 runsGroup 7: 4 methods × 2 (QAT on/off) × 3 seeds = 24 runsGroup 8: 4 sizes × 3 configs (s=.25, s=.50, joint) × 3 = 36 runs- Group 9: evaluation only (reuses checkpoints) = 0 runs+ Group 9: 3 act × (3 dense + 6 sparse + 3 joint) × 3 = 36 runs+ + 2 act × 3 seeds (large validation) = 6 runs+ Group 10: evaluation only (reuses checkpoints) = 0 runs─────────────────────────────────────────────────────────────────────- Total: 177 runs+ Total: 219 runs-At ~12 min/run: ~35 TPU-hours (very manageable on v5litepod-4)+At ~12 min/run: ~44 TPU-hours (very manageable on v5litepod-4)================================================================================SECTION 7 — ABLATION CHECKLIST@@ -363,6 +427,8 @@ Before submission, verify these are addressed:[ ] Pruning bias: report which layers (enc vs dec) get pruned[ ] Generation quality: not just perplexity, show actual text samples[ ] Group size sensitivity: optimal g, and does it interact with model scale?+ [ ] Activation × pruning: does dReLU help or hurt compression robustness?+ [ ] Activation sparsity measured: connect weight sparsity to activation sparsity[ ] Honest limitations: sparse matmuls don't accelerate on current JAX================================================================================@@ -381,7 +447,11 @@ Phase 3 (Day 3): Groups 5 + 6 + 7 + 8 (84 runs, ~17 hrs)Ablations: schedule, recovery, QAT interaction, group size.Immediate deliverable: complete ablation tables.-Phase 4 (Day 4): Group 9 + writing+Phase 4 (Day 4): Group 9 (42 runs, ~8 hrs)+ Activation × pruning interaction study.+ Immediate deliverable: activation robustness table + degradation plot.++Phase 5 (Day 5): Group 10 + writingRun inference benchmarks on all Pareto-optimal checkpoints.Begin paper draft with all tables and figures populated.diff --git a/src/cli.py b/src/cli.pyindex 51e8446..962879c 100644--- a/src/cli.py+++ b/src/cli.py@@ -140,6 +140,7 @@ def main():p.add_argument("--sparsity-ratio", type=float, default=0.33)p.add_argument("--layer-prune-ratio", type=float, default=0.33)p.add_argument("--group-size", type=int, default=32)+ p.add_argument("--activation", type=str, default="drelu", choices=["drelu", "swiglu", "geglu"])p = sub.add_parser("run", add_help=False)p.add_argument("--checkpoint", type=str, required=True)diff --git a/src/model.py b/src/model.pyindex cc7b4bf..a445730 100644--- a/src/model.py+++ b/src/model.py@@ -43,6 +43,7 @@ class TransformerConfig:pad_token_id: int = 0rope_theta: float = 10000.0dtype: str = "bfloat16"+ activation: str = "drelu"@propertydef jax_dtype(self):@@ -126,12 +127,18 @@ class FeedForward(nn.Module):num_layers: intdropout_rate: float = 0.1dtype: jnp.dtype = jnp.bfloat16+ activation: str = "drelu"@nn.compactdef __call__(self, x, deterministic=True):gate = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="gate_proj")(x)up = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="up_proj")(x)- x = nn.silu(gate) * up+ if self.activation == "swiglu":+ x = nn.silu(gate) * up+ elif self.activation == "geglu":+ x = nn.gelu(gate) * up+ else: # drelu+ x = nn.relu(gate) * nn.relu(up)x = nn.Dense(self.d_model, dtype=self.dtype, use_bias=False, kernel_init=residual_init(self.num_layers), name="down_proj")(x)x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)return x@@ -145,6 +152,7 @@ class EncoderBlock(nn.Module):num_layers: intdropout_rate: float = 0.1dtype: jnp.dtype = jnp.bfloat16+ activation: str = "drelu"@nn.compactdef __call__(self, x, mask=None, rope=None, deterministic=True):@@ -158,7 +166,7 @@ class EncoderBlock(nn.Module):residual = xx = ZCRMSNorm(dtype=self.dtype)(x)- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dropout_rate, self.dtype)(x, deterministic=deterministic)+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dropout_rate, self.dtype, self.activation)(x, deterministic=deterministic)x = x + residualreturn x@@ -172,6 +180,7 @@ class DecoderBlock(nn.Module):num_layers: intdropout_rate: float = 0.1dtype: jnp.dtype = jnp.bfloat16+ activation: str = "drelu"@nn.compactdef __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, deterministic=True):@@ -195,7 +204,7 @@ class DecoderBlock(nn.Module):residual = xx = ZCRMSNorm(dtype=self.dtype)(x)- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dropout_rate, self.dtype)(x, deterministic=deterministic)+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dropout_rate, self.dtype, self.activation)(x, deterministic=deterministic)x = x + residualreturn x@@ -213,7 +222,7 @@ class Encoder(nn.Module):for i in range(cfg.num_encoder_layers):x = EncoderBlock(- cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, cfg.dropout_rate, dt, name=f"block_{i}"+ cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, cfg.dropout_rate, dt, cfg.activation, name=f"block_{i}")(x, mask=mask, rope=rope, deterministic=deterministic)x = ZCRMSNorm(dtype=dt)(x)@@ -232,7 +241,7 @@ class Decoder(nn.Module):for i in range(cfg.num_decoder_layers):x = DecoderBlock(- cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, cfg.dropout_rate, dt, name=f"block_{i}"+ cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, cfg.dropout_rate, dt, cfg.activation, name=f"block_{i}")(x, encoder_out, self_mask=self_mask, cross_mask=cross_mask, rope=rope, deterministic=deterministic)x = ZCRMSNorm(dtype=dt)(x)diff --git a/src/train.py b/src/train.pyindex 8f65ee9..bac7d8d 100644--- a/src/train.py+++ b/src/train.py@@ -392,6 +392,7 @@ def train(args):max_seq_len=max(args.max_enc_len, args.max_dec_len),dropout_rate=args.dropout,dtype=args.dtype,+ activation=getattr(args, "activation", "drelu"),)# Set group size for QAT and sparsification (captured by _train_step closure)@@ -572,12 +573,36 @@ def train(args):epoch_avg_loss = sum(losses) / len(losses) if losses else float("nan")final_loss = losses[-1] if losses else float("nan")- print(f"Epoch {epoch + 1}/{args.epochs} — avg loss: {epoch_avg_loss:.4f}, final loss: {final_loss:.4f}")+ final_ppl = math.exp(final_loss) if not math.isnan(final_loss) else float("nan")++ # Compute val perplexity at end of epoch+ eval_params = jax_utils.unreplicate(ema_params)+ val_causal = make_causal_mask(args.max_dec_len)+ total_loss, total_toks = 0.0, 0.0+ for vb in get_batches(val_enc, val_dec_in, val_dec_tgt, args.batch_size, shuffle=False):+ vl, vt = val_loss_fn(eval_params, vb[0], vb[1], vb[2], val_causal)+ total_loss += float(vl)+ total_toks += float(vt)+ last_val_ppl = float(math.exp(total_loss / max(total_toks, 1)))++ epoch_params = jax.tree.map(np.array, eval_params)+ total_params = sum(x.size for x in jax.tree.leaves(epoch_params))+ near_zero = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(epoch_params))+ sparsity = near_zero / total_params * 100++ print(f"\n Epoch {epoch + 1}/{args.epochs} report:")+ print(f" Avg loss: {epoch_avg_loss:.4f}")+ print(f" Final loss: {final_loss:.4f}")+ print(f" Perplexity: {final_ppl:.2f}")+ print(f" Val perplexity: {last_val_ppl:.2f}")+ print(f" Weight sparsity: {sparsity:.2f}% ({near_zero:,}/{total_params:,} near-zero)")if use_wandb:wandb.log({"epoch/avg_loss": epoch_avg_loss,"epoch/final_loss": final_loss,+ "epoch/val_ppl": last_val_ppl,+ "epoch/weight_sparsity": sparsity,"epoch": epoch + 1,})@@ -589,27 +614,9 @@ def train(args):pickle.dump({"params": params_np, "config": config.__dict__}, f)print(f"Saved checkpoint: {ckpt_path}")- final_params = jax.tree.map(np.array, jax_utils.unreplicate(ema_params))- total_params = sum(x.size for x in jax.tree.leaves(final_params))- near_zero = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(final_params))- sparsity = near_zero / total_params * 100-- final_loss = losses[-1] if losses else float("nan")- final_ppl = math.exp(final_loss) if not math.isnan(final_loss) else float("nan")-- print("\n" + "=" * 50)- print("TRAINING REPORT")- print("=" * 50)- print(f" Final loss: {final_loss:.4f}")- print(f" Final perplexity: {final_ppl:.2f}")- print(f" Val perplexity: {last_val_ppl:.2f}" if last_val_ppl is not None else " Val perplexity: N/A")- print(f" Weight sparsity: {sparsity:.2f}% ({near_zero:,}/{total_params:,} near-zero)")- print("=" * 50)-if use_wandb:- wandb.log({"final/loss": final_loss, "final/ppl": final_ppl, "final/weight_sparsity": sparsity})wandb.finish()- print("Training complete.")+ print("\nTraining complete.")def sweep(args):