Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

Sparsification

HenryNdubuaku7ccad3128e45
diff --git a/experiements/pruning.txt b/experiements/pruning.txt
index 2a5f8e9..a3ca264 100644
--- a/experiements/pruning.txt
+++ b/experiements/pruning.txt
@@ -19,8 +19,9 @@ Independent variables:
B. Layer prune ratio l ∈ {0.0, 0.125, 0.25, 0.375, 0.50}
C. Group size g ∈ {16, 32, 64, 128}
D. Pruning schedule ∈ {staged, simultaneous, reverse-order}
- E. Model scale ∈ {BASE-256, LARGE-512}
- F. QAT ∈ {off, int4-group32}
+ E. FFN activation ∈ {dReLU, SwiGLU, GeGLU}
+ F. Model scale ∈ {BASE-256, LARGE-512}
+ G. QAT ∈ {off, int4-group32}
Dependent variables (measured):
- Validation perplexity (TinyStories)
@@ -263,7 +264,65 @@ Deliverables:
bits and 32/g is amortized scale overhead per weight.
-------------------------------------------------------------------------------
-GROUP 9: Inference Efficiency Validation (Table 8, Figure 7)
+GROUP 9: Activation Function × Pruning Interaction (Table 8, Figure 8)
+ *** KEY FINDING ***
+-------------------------------------------------------------------------------
+Purpose: Test whether activation function choice determines pruning robustness.
+Early observation: dReLU dramatically increases post-prune perplexity compared
+to SwiGLU. This group systematically characterises the interaction.
+
+Activation definitions:
+ - dReLU: ReLU(gate) * ReLU(up) — naturally ~75% sparse activations
+ - SwiGLU: SiLU(gate) * up — smooth, dense activations
+ - GeGLU: GELU(gate) * up — smooth, nearly dense activations
+
+Requires: --activation CLI flag wired to FeedForward module.
+
+ 9a. Dense baselines (no pruning) — isolate activation quality
+ For act in {drelu, swiglu, geglu}:
+ needle train --epochs 3 --activation {act} --sparsity-ratio 0.0 \
+ --layer-prune-ratio 0.0 --seed {42,137,256}
+
+ 9b. Sparsification only — test pruning robustness per activation
+ For act in {drelu, swiglu, geglu}, s in {0.25, 0.50}:
+ needle train --epochs 3 --activation {act} --sparsity-ratio {s} \
+ --seed {42,137,256}
+
+ 9c. Joint sparsification + layer pruning — full compression
+ For act in {drelu, swiglu, geglu}:
+ needle train --epochs 4 --activation {act} --sparsity-ratio 0.25 \
+ --layer-prune-ratio 0.25 --seed {42,137,256}
+
+ 9d. LARGE-512 validation on best and worst activation from 9b-9c
+ needle train --epochs 4 --d-model 512 --num-heads 8 --num-layers 8 \
+ --activation {act} --sparsity-ratio 0.25 --layer-prune-ratio 0.25 \
+ --seed {42,137,256}
+
+Deliverables:
+ - Grouped bar chart: activation × {dense, s=0.25, s=0.50} → val ppl (Fig 8a)
+ - Table: full metrics for all activation × compression combos
+ - Relative degradation plot: (pruned_ppl / dense_ppl) per activation (Fig 8b)
+ — THIS IS THE KEY CHART: shows pruning *robustness* not just raw quality
+ - Analysis of competing hypotheses:
+ (a) dReLU hurts pruning because its ~75% activation sparsity means fewer
+ active paths compensate for zeroed weights — the network has less
+ redundancy to absorb pruning damage
+ (b) dReLU helps pruning because its natural sparsity pattern means weight
+ blocks that process mostly-zero activations are truly unimportant
+ and can be safely pruned (better signal for the pruning criterion)
+ (c) Smooth activations (SwiGLU/GeGLU) spread information across more
+ weights, creating redundancy that makes the model more robust to
+ weight removal but harder to compress efficiently
+ - Recommendation: which activation to use when pruning is planned vs not
+
+ Additional metric for this group:
+ - Activation sparsity: measure % of zeros in FFN intermediate activations
+ for each activation function at each pruning level. Report as:
+ act_sparsity = E[fraction of zeros in ReLU(gate) * ReLU(up)]
+ This connects weight sparsity to activation sparsity.
+
+-------------------------------------------------------------------------------
+GROUP 10: Inference Efficiency Validation (Table 9, Figure 9)
-------------------------------------------------------------------------------
Purpose: Translate theoretical compression into real speedup.
@@ -315,7 +374,8 @@ Tables:
T5. Schedule ablation (3 schedules)
T6. QAT interaction (4 methods × {on,off})
T7. Group size ablation (4 sizes × 2 sparsity levels + joint)
- T8. Inference efficiency (Pareto-optimal checkpoints)
+ T8. Activation × pruning interaction (3 activations × 4 compression levels)
+ T9. Inference efficiency (Pareto-optimal checkpoints)
Figures:
F1. Sparsity-only Pareto: sparsity % vs perplexity + LAMBADA
@@ -325,6 +385,8 @@ Figures:
F5. Compression ratio vs actual inference speedup
F6. Group size: (a) g vs perplexity at s=0.25/0.50, (b) g vs LAMBADA
F7. Effective bits-per-param vs quality across group sizes
+ F8. Activation × pruning: (a) grouped bars, (b) relative degradation ratio
+ F9. Activation sparsity vs weight sparsity interaction
================================================================================
SECTION 6 — COMPUTE BUDGET
@@ -342,11 +404,13 @@ Estimated total runs:
Group 6: 5 epoch counts × 3 seeds = 15 runs
Group 7: 4 methods × 2 (QAT on/off) × 3 seeds = 24 runs
Group 8: 4 sizes × 3 configs (s=.25, s=.50, joint) × 3 = 36 runs
- Group 9: evaluation only (reuses checkpoints) = 0 runs
+ Group 9: 3 act × (3 dense + 6 sparse + 3 joint) × 3 = 36 runs
+ + 2 act × 3 seeds (large validation) = 6 runs
+ Group 10: evaluation only (reuses checkpoints) = 0 runs
─────────────────────────────────────────────────────────────────────
- Total: 177 runs
+ Total: 219 runs
-At ~12 min/run: ~35 TPU-hours (very manageable on v5litepod-4)
+At ~12 min/run: ~44 TPU-hours (very manageable on v5litepod-4)
================================================================================
SECTION 7 — ABLATION CHECKLIST
@@ -363,6 +427,8 @@ Before submission, verify these are addressed:
[ ] Pruning bias: report which layers (enc vs dec) get pruned
[ ] Generation quality: not just perplexity, show actual text samples
[ ] Group size sensitivity: optimal g, and does it interact with model scale?
+ [ ] Activation × pruning: does dReLU help or hurt compression robustness?
+ [ ] Activation sparsity measured: connect weight sparsity to activation sparsity
[ ] Honest limitations: sparse matmuls don't accelerate on current JAX
================================================================================
@@ -381,7 +447,11 @@ Phase 3 (Day 3): Groups 5 + 6 + 7 + 8 (84 runs, ~17 hrs)
Ablations: schedule, recovery, QAT interaction, group size.
Immediate deliverable: complete ablation tables.
-Phase 4 (Day 4): Group 9 + writing
+Phase 4 (Day 4): Group 9 (42 runs, ~8 hrs)
+ Activation × pruning interaction study.
+ Immediate deliverable: activation robustness table + degradation plot.
+
+Phase 5 (Day 5): Group 10 + writing
Run inference benchmarks on all Pareto-optimal checkpoints.
Begin paper draft with all tables and figures populated.
diff --git a/src/cli.py b/src/cli.py
index 51e8446..962879c 100644
--- a/src/cli.py
+++ b/src/cli.py
@@ -140,6 +140,7 @@ def main():
p.add_argument("--sparsity-ratio", type=float, default=0.33)
p.add_argument("--layer-prune-ratio", type=float, default=0.33)
p.add_argument("--group-size", type=int, default=32)
+ p.add_argument("--activation", type=str, default="drelu", choices=["drelu", "swiglu", "geglu"])
p = sub.add_parser("run", add_help=False)
p.add_argument("--checkpoint", type=str, required=True)
diff --git a/src/model.py b/src/model.py
index cc7b4bf..a445730 100644
--- a/src/model.py
+++ b/src/model.py
@@ -43,6 +43,7 @@ class TransformerConfig:
pad_token_id: int = 0
rope_theta: float = 10000.0
dtype: str = "bfloat16"
+ activation: str = "drelu"
@property
def jax_dtype(self):
@@ -126,12 +127,18 @@ class FeedForward(nn.Module):
num_layers: int
dropout_rate: float = 0.1
dtype: jnp.dtype = jnp.bfloat16
+ activation: str = "drelu"
@nn.compact
def __call__(self, x, deterministic=True):
gate = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="gate_proj")(x)
up = nn.Dense(self.d_ff, dtype=self.dtype, use_bias=False, kernel_init=default_init(), name="up_proj")(x)
- x = nn.silu(gate) * up
+ if self.activation == "swiglu":
+ x = nn.silu(gate) * up
+ elif self.activation == "geglu":
+ x = nn.gelu(gate) * up
+ else: # drelu
+ x = nn.relu(gate) * nn.relu(up)
x = nn.Dense(self.d_model, dtype=self.dtype, use_bias=False, kernel_init=residual_init(self.num_layers), name="down_proj")(x)
x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)
return x
@@ -145,6 +152,7 @@ class EncoderBlock(nn.Module):
num_layers: int
dropout_rate: float = 0.1
dtype: jnp.dtype = jnp.bfloat16
+ activation: str = "drelu"
@nn.compact
def __call__(self, x, mask=None, rope=None, deterministic=True):
@@ -158,7 +166,7 @@ class EncoderBlock(nn.Module):
residual = x
x = ZCRMSNorm(dtype=self.dtype)(x)
- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dropout_rate, self.dtype)(x, deterministic=deterministic)
+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dropout_rate, self.dtype, self.activation)(x, deterministic=deterministic)
x = x + residual
return x
@@ -172,6 +180,7 @@ class DecoderBlock(nn.Module):
num_layers: int
dropout_rate: float = 0.1
dtype: jnp.dtype = jnp.bfloat16
+ activation: str = "drelu"
@nn.compact
def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, deterministic=True):
@@ -195,7 +204,7 @@ class DecoderBlock(nn.Module):
residual = x
x = ZCRMSNorm(dtype=self.dtype)(x)
- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dropout_rate, self.dtype)(x, deterministic=deterministic)
+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dropout_rate, self.dtype, self.activation)(x, deterministic=deterministic)
x = x + residual
return x
@@ -213,7 +222,7 @@ class Encoder(nn.Module):
for i in range(cfg.num_encoder_layers):
x = EncoderBlock(
- cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, cfg.dropout_rate, dt, name=f"block_{i}"
+ cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, cfg.dropout_rate, dt, cfg.activation, name=f"block_{i}"
)(x, mask=mask, rope=rope, deterministic=deterministic)
x = ZCRMSNorm(dtype=dt)(x)
@@ -232,7 +241,7 @@ class Decoder(nn.Module):
for i in range(cfg.num_decoder_layers):
x = DecoderBlock(
- cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, cfg.dropout_rate, dt, name=f"block_{i}"
+ cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff, cfg.total_layers, cfg.dropout_rate, dt, cfg.activation, name=f"block_{i}"
)(x, encoder_out, self_mask=self_mask, cross_mask=cross_mask, rope=rope, deterministic=deterministic)
x = ZCRMSNorm(dtype=dt)(x)
diff --git a/src/train.py b/src/train.py
index 8f65ee9..bac7d8d 100644
--- a/src/train.py
+++ b/src/train.py
@@ -392,6 +392,7 @@ def train(args):
max_seq_len=max(args.max_enc_len, args.max_dec_len),
dropout_rate=args.dropout,
dtype=args.dtype,
+ activation=getattr(args, "activation", "drelu"),
)
# Set group size for QAT and sparsification (captured by _train_step closure)
@@ -572,12 +573,36 @@ def train(args):
epoch_avg_loss = sum(losses) / len(losses) if losses else float("nan")
final_loss = losses[-1] if losses else float("nan")
- print(f"Epoch {epoch + 1}/{args.epochs} — avg loss: {epoch_avg_loss:.4f}, final loss: {final_loss:.4f}")
+ final_ppl = math.exp(final_loss) if not math.isnan(final_loss) else float("nan")
+
+ # Compute val perplexity at end of epoch
+ eval_params = jax_utils.unreplicate(ema_params)
+ val_causal = make_causal_mask(args.max_dec_len)
+ total_loss, total_toks = 0.0, 0.0
+ for vb in get_batches(val_enc, val_dec_in, val_dec_tgt, args.batch_size, shuffle=False):
+ vl, vt = val_loss_fn(eval_params, vb[0], vb[1], vb[2], val_causal)
+ total_loss += float(vl)
+ total_toks += float(vt)
+ last_val_ppl = float(math.exp(total_loss / max(total_toks, 1)))
+
+ epoch_params = jax.tree.map(np.array, eval_params)
+ total_params = sum(x.size for x in jax.tree.leaves(epoch_params))
+ near_zero = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(epoch_params))
+ sparsity = near_zero / total_params * 100
+
+ print(f"\n Epoch {epoch + 1}/{args.epochs} report:")
+ print(f" Avg loss: {epoch_avg_loss:.4f}")
+ print(f" Final loss: {final_loss:.4f}")
+ print(f" Perplexity: {final_ppl:.2f}")
+ print(f" Val perplexity: {last_val_ppl:.2f}")
+ print(f" Weight sparsity: {sparsity:.2f}% ({near_zero:,}/{total_params:,} near-zero)")
if use_wandb:
wandb.log({
"epoch/avg_loss": epoch_avg_loss,
"epoch/final_loss": final_loss,
+ "epoch/val_ppl": last_val_ppl,
+ "epoch/weight_sparsity": sparsity,
"epoch": epoch + 1,
})
@@ -589,27 +614,9 @@ def train(args):
pickle.dump({"params": params_np, "config": config.__dict__}, f)
print(f"Saved checkpoint: {ckpt_path}")
- final_params = jax.tree.map(np.array, jax_utils.unreplicate(ema_params))
- total_params = sum(x.size for x in jax.tree.leaves(final_params))
- near_zero = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(final_params))
- sparsity = near_zero / total_params * 100
-
- final_loss = losses[-1] if losses else float("nan")
- final_ppl = math.exp(final_loss) if not math.isnan(final_loss) else float("nan")
-
- print("\n" + "=" * 50)
- print("TRAINING REPORT")
- print("=" * 50)
- print(f" Final loss: {final_loss:.4f}")
- print(f" Final perplexity: {final_ppl:.2f}")
- print(f" Val perplexity: {last_val_ppl:.2f}" if last_val_ppl is not None else " Val perplexity: N/A")
- print(f" Weight sparsity: {sparsity:.2f}% ({near_zero:,}/{total_params:,} near-zero)")
- print("=" * 50)
-
if use_wandb:
- wandb.log({"final/loss": final_loss, "final/ppl": final_ppl, "final/weight_sparsity": sparsity})
wandb.finish()
- print("Training complete.")
+ print("\nTraining complete.")
def sweep(args):