Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

pruning techniques

HenryNdubuaku3940814ea1a1
diff --git a/experiements/pruning.txt b/experiements/pruning.txt
new file mode 100644
index 0000000..2a5f8e9
--- /dev/null
+++ b/experiements/pruning.txt
@@ -0,0 +1,388 @@
+================================================================================
+STRUCTURED COMPRESSION OF SMALL ENCODER-DECODER TRANSFORMERS:
+JOINT GROUP-WISE SPARSIFICATION AND LAYER PRUNING
+================================================================================
+Target venue: ICML 2026
+
+Thesis: Structured compression (group-aligned block sparsity + layer removal)
+applied in a staged schedule during training yields Pareto-optimal accuracy-
+efficiency tradeoffs for small encoder-decoder models, matching or exceeding
+post-hoc pruning at equivalent FLOPs reduction while preserving INT4
+quantization compatibility.
+
+================================================================================
+SECTION 1 — EXPERIMENTAL AXES
+================================================================================
+
+Independent variables:
+ A. Sparsity ratio s ∈ {0.0, 0.10, 0.25, 0.40, 0.50}
+ B. Layer prune ratio l ∈ {0.0, 0.125, 0.25, 0.375, 0.50}
+ C. Group size g ∈ {16, 32, 64, 128}
+ D. Pruning schedule ∈ {staged, simultaneous, reverse-order}
+ E. Model scale ∈ {BASE-256, LARGE-512}
+ F. QAT ∈ {off, int4-group32}
+
+Dependent variables (measured):
+ - Validation perplexity (TinyStories)
+ - WikiText-2 perplexity
+ - LAMBADA accuracy
+ - HellaSwag accuracy
+ - ARC-Easy accuracy
+ - Throughput (tokens/sec on inference)
+ - Bigram repetition rate (generation diversity)
+ - Parameter count (absolute and % of dense baseline)
+ - Effective FLOPs (params * density)
+ - Weight sparsity (% near-zero, measured at threshold < 1e-6)
+
+================================================================================
+SECTION 2 — MODEL CONFIGURATIONS
+================================================================================
+
+BASE-256 (25.4M params, default):
+ d_model=256, num_heads=4, num_kv_heads=2, num_layers=4,
+ d_ff=1024, max_seq_len=256, vocab=65536, dtype=bfloat16
+
+LARGE-512 (≈100M params, scaling study):
+ d_model=512, num_heads=8, num_kv_heads=4, num_layers=8,
+ d_ff=2048, max_seq_len=256, vocab=65536, dtype=bfloat16
+
+Both trained on TinyStories (500K samples, encoder-decoder split at 30%).
+Optimizer: Muon (kernels) + AdamW (embeddings/norms), WSD schedule.
+
+================================================================================
+SECTION 3 — EXPERIMENT GROUPS
+================================================================================
+
+All experiments use 3 seeds (42, 137, 256) for statistical significance.
+Report mean ± std on all metrics.
+
+-------------------------------------------------------------------------------
+GROUP 1: Dense Baselines (Table 1)
+-------------------------------------------------------------------------------
+Purpose: Establish upper-bound quality for each model scale.
+
+ 1a. BASE-256, 3 epochs, no pruning, no QAT
+ needle train --epochs 3 --seed {42,137,256}
+
+ 1b. BASE-256, 3 epochs, no pruning, with QAT (int4-group32)
+ [QAT is always on in current code — this IS the baseline]
+
+ 1c. LARGE-512, 3 epochs, no pruning
+ needle train --epochs 3 --d-model 512 --num-heads 8 --num-layers 8 \
+ --max-enc-len 256 --max-dec-len 256 --seed {42,137,256}
+
+Deliverables:
+ - Val perplexity, benchmark scores, throughput, generation samples
+ - These are the denominators for all relative comparisons
+
+-------------------------------------------------------------------------------
+GROUP 2: Group-Wise Sparsification Only (Table 2, Figure 1)
+-------------------------------------------------------------------------------
+Purpose: Characterise accuracy-sparsity Pareto frontier for block pruning alone.
+Schedule: Epoch 1 dense, epochs 2-3 with sparsity mask locked.
+
+ For s in {0.10, 0.25, 0.40, 0.50}:
+ 2a. BASE-256, 3 epochs, --sparsity-ratio s
+ needle train --epochs 3 --sparsity-ratio {s} --seed {42,137,256}
+
+ For s in {0.10, 0.25, 0.40, 0.50}:
+ 2b. LARGE-512, 3 epochs, --sparsity-ratio s
+ needle train --epochs 3 --d-model 512 --num-heads 8 --num-layers 8 \
+ --sparsity-ratio {s} --seed {42,137,256}
+
+Deliverables:
+ - Pareto curve: sparsity % vs val perplexity (Figure 1a)
+ - Pareto curve: sparsity % vs LAMBADA accuracy (Figure 1b)
+ - Table: all metrics at each sparsity level
+ - Analysis: at what sparsity does quality degrade >5% relative to baseline?
+
+-------------------------------------------------------------------------------
+GROUP 3: Layer Pruning Only (Table 3, Figure 2)
+-------------------------------------------------------------------------------
+Purpose: Characterise accuracy-compression frontier for layer removal alone.
+Schedule: Epoch 1 dense, epochs 2-3 with reduced architecture.
+
+ For l in {0.125, 0.25, 0.375, 0.50}:
+ 3a. BASE-256 (4+4 layers), 3 epochs, --layer-prune-ratio l
+ needle train --epochs 3 --layer-prune-ratio {l} --seed {42,137,256}
+ Expected surviving layers: 7, 6, 5, 4 (of 8 total)
+
+ For l in {0.125, 0.25, 0.375, 0.50}:
+ 3b. LARGE-512 (8+8 layers), 3 epochs, --layer-prune-ratio l
+ needle train --epochs 3 --d-model 512 --num-heads 8 --num-layers 8 \
+ --layer-prune-ratio {l} --seed {42,137,256}
+ Expected surviving layers: 14, 12, 10, 8 (of 16 total)
+
+Deliverables:
+ - Pareto curve: layer ratio vs val perplexity (Figure 2a)
+ - Pareto curve: parameter count vs LAMBADA accuracy (Figure 2b)
+ - Table: per-run encoder/decoder surviving layer counts
+ - Analysis: does the pruner prefer removing encoder or decoder layers?
+ (Hypothesis: encoder layers are pruned more aggressively since the
+ decoder carries more of the generation burden.)
+
+-------------------------------------------------------------------------------
+GROUP 4: Joint Sparsification + Layer Pruning (Table 4, Figure 3)
+ *** CORE CONTRIBUTION ***
+-------------------------------------------------------------------------------
+Purpose: Show that combining both methods outperforms either alone at matched
+compression ratio. This is the central claim of the paper.
+
+Schedule (staged):
+ Epoch 1: dense training
+ Epoch 2: apply group-wise sparsification, train with mask
+ Epoch 3: apply layer pruning, train with reduced architecture + mask
+ Epoch 4: recovery fine-tuning
+
+ For (s, l) in grid:
+ {(0.10, 0.125), (0.10, 0.25),
+ (0.25, 0.125), (0.25, 0.25), (0.25, 0.375),
+ (0.40, 0.125), (0.40, 0.25),
+ (0.50, 0.25), (0.50, 0.50)}
+
+ 4a. BASE-256, 4 epochs
+ needle train --epochs 4 --sparsity-ratio {s} --layer-prune-ratio {l} \
+ --seed {42,137,256}
+
+ 4b. LARGE-512, select best 4 (s,l) pairs from 4a, 4 epochs
+ needle train --epochs 4 --d-model 512 --num-heads 8 --num-layers 8 \
+ --sparsity-ratio {s} --layer-prune-ratio {l} --seed {42,137,256}
+
+Deliverables:
+ - 2D heatmap: (s, l) → val perplexity (Figure 3a)
+ - 2D heatmap: (s, l) → effective parameter count (Figure 3b)
+ - Pareto overlay: joint vs sparsity-only vs layer-only at matched
+ compression (Figure 3c) — THIS IS THE KEY FIGURE
+ - Table: full metrics for all (s, l) pairs
+ - Analysis: multiplicative vs additive compression benefit
+
+-------------------------------------------------------------------------------
+GROUP 5: Schedule Ablation (Table 5)
+-------------------------------------------------------------------------------
+Purpose: Justify the staged schedule (sparsify first, then prune layers).
+
+Pick the best (s, l) pair from Group 4a. Run 3 schedules:
+
+ 5a. Staged (default): sparsify epoch 2, layer prune epoch 3, recover epoch 4
+ needle train --epochs 4 --sparsity-ratio {s} --layer-prune-ratio {l}
+
+ 5b. Reverse: layer prune epoch 2, sparsify epoch 3, recover epoch 4
+ [Requires code change to swap epoch triggers]
+
+ 5c. Simultaneous: both at epoch 2, recover epochs 3-4
+ [Requires code change to apply both at same epoch]
+
+Deliverables:
+ - Table: val perplexity and benchmark scores for each schedule
+ - Analysis: why staged order matters (hypothesis: sparsification is
+ lower-risk, lets model adapt before the more destructive layer removal)
+
+-------------------------------------------------------------------------------
+GROUP 6: Recovery Epoch Scaling (Figure 4)
+-------------------------------------------------------------------------------
+Purpose: How many recovery epochs are needed after compression?
+
+Pick the best (s, l) pair from Group 4a. Vary total epochs:
+
+ For E in {2, 3, 4, 5, 6}:
+ 6a. BASE-256, E epochs, --sparsity-ratio {s} --layer-prune-ratio {l}
+ (pruning always at epochs 2 and 3; recovery = E - 3 epochs)
+
+Deliverables:
+ - Line plot: recovery epochs vs val perplexity (Figure 4)
+ - Analysis: diminishing returns curve, recommend minimum recovery budget
+
+-------------------------------------------------------------------------------
+GROUP 7: QAT Interaction Study (Table 6)
+-------------------------------------------------------------------------------
+Purpose: Show that group-wise sparsification is compatible with INT4 QAT
+because both operate on the same group_size=32 granularity.
+
+ 7a. Dense + QAT (baseline, = Group 1b)
+ 7b. Sparsified (s=0.25) + QAT
+ 7c. Layer-pruned (l=0.25) + QAT
+ 7d. Joint (s=0.25, l=0.25) + QAT
+
+ For comparison, also run 7b-7d WITHOUT QAT by modifying _train_step
+ to not call _quantize_params.
+
+Deliverables:
+ - Table: QAT on vs off for each compression method
+ - Analysis: does QAT hurt more when combined with sparsification?
+ (Hypothesis: no, because zeroed groups have no quantization error)
+ - Effective compression ratio: sparsity + INT4 → bits per parameter
+
+-------------------------------------------------------------------------------
+GROUP 8: Group Size Ablation (Table 7, Figure 6)
+-------------------------------------------------------------------------------
+Purpose: Determine optimal granularity for block sparsification. Smaller groups
+give finer-grained pruning decisions (better accuracy retention) but produce
+less structured sparsity (harder to exploit in hardware). Larger groups are
+more hardware-friendly but coarser. The group size also controls QAT
+quantization granularity, so this directly tests the sparsity-quantization
+co-design hypothesis.
+
+Note: requires adding --group-size CLI flag and threading it through to
+_block_prune_params and _fake_quantize_int4.
+
+Fixed configuration: BASE-256, s=0.25, 3 epochs, QAT on.
+
+ For g in {16, 32, 64, 128}:
+ 8a. Sparsification only (isolate group size effect on pruning)
+ needle train --epochs 3 --sparsity-ratio 0.25 --group-size {g} \
+ --seed {42,137,256}
+
+ 8b. Joint sparsification + layer pruning (best l from Group 4a)
+ needle train --epochs 4 --sparsity-ratio 0.25 --layer-prune-ratio {l} \
+ --group-size {g} --seed {42,137,256}
+
+Additional comparison at high sparsity to stress-test granularity:
+
+ For g in {16, 32, 64, 128}:
+ 8c. s=0.50, sparsification only
+ needle train --epochs 3 --sparsity-ratio 0.50 --group-size {g} \
+ --seed {42,137,256}
+
+Deliverables:
+ - Line plot: group size vs val perplexity at s=0.25 and s=0.50 (Figure 6a)
+ - Line plot: group size vs LAMBADA accuracy (Figure 6b)
+ - Table: all metrics at each group size for both sparsity levels
+ - Analysis of key tradeoffs:
+ (a) g=16: finest granularity, best accuracy, but 2× more scale factors
+ to store than g=32 and least structured pattern
+ (b) g=32: default, matches common INT4 quantization group size,
+ good balance of accuracy and structure
+ (c) g=64: coarser, each pruning decision covers 64 weights,
+ more aggressive but more hardware-friendly
+ (d) g=128: coarsest, each block is a full row for d_model=128,
+ approaches per-channel granularity, may degrade significantly
+ - Cross-reference with QAT: when group_size is shared between sparsification
+ and quantization, zeroed groups save both weight storage AND scale storage
+ (no scale needed for an all-zero group). Report effective bits-per-param
+ accounting for this: bpp = (1 - sparsity) × (4 + 32/g) where 4 is INT4
+ bits and 32/g is amortized scale overhead per weight.
+
+-------------------------------------------------------------------------------
+GROUP 9: Inference Efficiency Validation (Table 8, Figure 7)
+-------------------------------------------------------------------------------
+Purpose: Translate theoretical compression into real speedup.
+
+For each Pareto-optimal checkpoint from Groups 2-4:
+ needle test --checkpoint {path} --throughput-runs 50
+
+Measure:
+ - Tokens/sec (decode throughput)
+ - Latency per generation (ms)
+ - Peak memory (via JAX profiler)
+
+Deliverables:
+ - Scatter plot: compression ratio vs actual speedup (Figure 5)
+ - Table: all Pareto-optimal models with throughput + quality
+ - Analysis: does structured sparsity translate to real speedup?
+ (Note: dense JAX matmuls don't skip zeros — speedup comes from
+ layer removal. Sparsity benefit is in model size / memory only
+ until sparse kernel support is added.)
+
+================================================================================
+SECTION 4 — EVALUATION PROTOCOL
+================================================================================
+
+For every checkpoint:
+
+ 1. Validation perplexity (TinyStories val split, full)
+ needle test --checkpoint {path} --max-eval-samples 5000
+
+ 2. Benchmark suite
+ needle evaluate --checkpoint {path} \
+ --benchmarks wikitext2 lambada hellaswag arc_easy --max-samples 500
+
+ 3. Generation quality (qualitative + bigram diversity)
+ needle test --checkpoint {path}
+ → Inspect 10 sample generations, report bigram repetition rate
+
+ 4. Throughput measurement
+ needle test --checkpoint {path} --throughput-runs 50
+
+================================================================================
+SECTION 5 — EXPECTED TABLES AND FIGURES
+================================================================================
+
+Tables:
+ T1. Dense baselines (2 scales × {with,without} QAT)
+ T2. Group-wise sparsification sweep (2 scales × 4 ratios)
+ T3. Layer pruning sweep (2 scales × 4 ratios)
+ T4. Joint compression grid (9 pairs × 2 scales)
+ T5. Schedule ablation (3 schedules)
+ T6. QAT interaction (4 methods × {on,off})
+ T7. Group size ablation (4 sizes × 2 sparsity levels + joint)
+ T8. Inference efficiency (Pareto-optimal checkpoints)
+
+Figures:
+ F1. Sparsity-only Pareto: sparsity % vs perplexity + LAMBADA
+ F2. Layer-prune-only Pareto: param count vs perplexity + LAMBADA
+ F3. Joint compression: (a) heatmap, (b) param heatmap, (c) Pareto overlay
+ F4. Recovery epoch scaling curve
+ F5. Compression ratio vs actual inference speedup
+ F6. Group size: (a) g vs perplexity at s=0.25/0.50, (b) g vs LAMBADA
+ F7. Effective bits-per-param vs quality across group sizes
+
+================================================================================
+SECTION 6 — COMPUTE BUDGET
+================================================================================
+
+Per-run cost (BASE-256, 4 epochs, 500K samples, 4× TPU v5 lite):
+ ~10 minutes training + ~2 minutes evaluation
+
+Estimated total runs:
+ Group 1: 2 configs × 3 seeds = 6 runs
+ Group 2: 4 ratios × 2 scales × 3 seeds = 24 runs
+ Group 3: 4 ratios × 2 scales × 3 seeds = 24 runs
+ Group 4: 9 pairs × 3 seeds + 4 pairs × 3 seeds (large) = 39 runs
+ Group 5: 3 schedules × 3 seeds = 9 runs
+ Group 6: 5 epoch counts × 3 seeds = 15 runs
+ Group 7: 4 methods × 2 (QAT on/off) × 3 seeds = 24 runs
+ Group 8: 4 sizes × 3 configs (s=.25, s=.50, joint) × 3 = 36 runs
+ Group 9: evaluation only (reuses checkpoints) = 0 runs
+ ─────────────────────────────────────────────────────────────────────
+ Total: 177 runs
+
+At ~12 min/run: ~35 TPU-hours (very manageable on v5litepod-4)
+
+================================================================================
+SECTION 7 — ABLATION CHECKLIST
+================================================================================
+
+Before submission, verify these are addressed:
+
+ [ ] Statistical significance: 3 seeds, report mean ± std
+ [ ] Matched comparisons: compare joint vs individual at SAME compression
+ [ ] Scaling behavior: BASE-256 and LARGE-512 show consistent trends
+ [ ] Schedule sensitivity: staged vs reverse vs simultaneous
+ [ ] Recovery saturation: diminishing returns on fine-tuning epochs
+ [ ] QAT compatibility: sparsity + quantization are not antagonistic
+ [ ] Pruning bias: report which layers (enc vs dec) get pruned
+ [ ] Generation quality: not just perplexity, show actual text samples
+ [ ] Group size sensitivity: optimal g, and does it interact with model scale?
+ [ ] Honest limitations: sparse matmuls don't accelerate on current JAX
+
+================================================================================
+SECTION 8 — EXECUTION ORDER
+================================================================================
+
+Phase 1 (Day 1): Groups 1 + 2 + 3 (54 runs, ~11 hrs)
+ Run all baselines and single-method sweeps in parallel across seeds.
+ Immediate deliverable: Pareto curves for each method alone.
+
+Phase 2 (Day 2): Group 4 (39 runs, ~8 hrs)
+ Joint compression grid. This produces the key result.
+ Immediate deliverable: heatmaps and overlay Pareto.
+
+Phase 3 (Day 3): Groups 5 + 6 + 7 + 8 (84 runs, ~17 hrs)
+ Ablations: schedule, recovery, QAT interaction, group size.
+ Immediate deliverable: complete ablation tables.
+
+Phase 4 (Day 4): Group 9 + writing
+ Run inference benchmarks on all Pareto-optimal checkpoints.
+ Begin paper draft with all tables and figures populated.
+
+================================================================================
diff --git a/src/cli.py b/src/cli.py
index d55465d..51e8446 100644
--- a/src/cli.py
+++ b/src/cli.py
@@ -137,7 +137,9 @@ def main():
p.add_argument("--seed", type=int, default=None)
p.add_argument("--eval-every", type=int, default=1000)
p.add_argument("--max-eval-samples", type=int, default=None)
- p.add_argument("--prune-ratio", type=float, default=0.0)
+ p.add_argument("--sparsity-ratio", type=float, default=0.33)
+ p.add_argument("--layer-prune-ratio", type=float, default=0.33)
+ p.add_argument("--group-size", type=int, default=32)
p = sub.add_parser("run", add_help=False)
p.add_argument("--checkpoint", type=str, required=True)
diff --git a/src/train.py b/src/train.py
index 35b2006..8f65ee9 100644
--- a/src/train.py
+++ b/src/train.py
@@ -204,16 +204,89 @@ def _block_prune_params(params, prune_ratio=0.25, group_size=32):
return jax.tree_util.tree_map_with_path(_prune_weight, params)
-def _quantize_params(params):
+def _layer_prune(params, config, layer_prune_ratio=0.25):
+ """Remove the lowest-scoring encoder/decoder blocks by L1 magnitude.
+
+ Returns (new_params, new_config) with surviving blocks renumbered.
+ """
+ from flax.core import unfreeze
+ p = unfreeze(params)
+
+ def _score_blocks(module_params, prefix):
+ scores = {}
+ for key, val in module_params.items():
+ if key.startswith("block_"):
+ s = sum(float(np.sum(np.abs(np.array(l)))) for l in jax.tree.leaves(val))
+ scores[key] = s
+ return scores
+
+ enc_scores = _score_blocks(p["encoder"], "encoder")
+ dec_scores = _score_blocks(p["decoder"], "decoder")
+
+ # Pool all block scores, find global threshold
+ all_scores = list(enc_scores.values()) + list(dec_scores.values())
+ if not all_scores:
+ return params, config
+
+ # Keep at least 1 encoder and 1 decoder block
+ num_enc = len(enc_scores)
+ num_dec = len(dec_scores)
+ total_to_remove = max(0, int(len(all_scores) * layer_prune_ratio))
+
+ # Score and rank all blocks together
+ tagged = [(s, "encoder", k) for k, s in enc_scores.items()] + \
+ [(s, "decoder", k) for k, s in dec_scores.items()]
+ tagged.sort(key=lambda x: x[0])
+
+ remove_enc = set()
+ remove_dec = set()
+ for score, module, key in tagged:
+ if len(remove_enc) + len(remove_dec) >= total_to_remove:
+ break
+ if module == "encoder" and (num_enc - len(remove_enc)) > 1:
+ remove_enc.add(key)
+ elif module == "decoder" and (num_dec - len(remove_dec)) > 1:
+ remove_dec.add(key)
+
+ # Rebuild with surviving blocks renumbered
+ def _keep_and_renumber(module_params, remove_set):
+ surviving = [(k, v) for k, v in sorted(module_params.items()) if k.startswith("block_") and k not in remove_set]
+ non_blocks = {k: v for k, v in module_params.items() if not k.startswith("block_")}
+ new = dict(non_blocks)
+ for i, (_, v) in enumerate(surviving):
+ new[f"block_{i}"] = v
+ return new
+
+ p["encoder"] = _keep_and_renumber(p["encoder"], remove_enc)
+ p["decoder"] = _keep_and_renumber(p["decoder"], remove_dec)
+
+ new_num_enc = num_enc - len(remove_enc)
+ new_num_dec = num_dec - len(remove_dec)
+ print(f" Encoder: {num_enc} -> {new_num_enc} blocks (removed {len(remove_enc)})")
+ print(f" Decoder: {num_dec} -> {new_num_dec} blocks (removed {len(remove_dec)})")
+
+ from dataclasses import replace as dc_replace
+ new_config = dc_replace(config,
+ num_encoder_layers=new_num_enc,
+ num_decoder_layers=new_num_dec)
+
+ return p, new_config
+
+
+def _quantize_params(params, group_size=32):
"""Fake-quantize all Dense kernels in the param tree."""
def _maybe_quantize(path, leaf):
name = path[-1].key if hasattr(path[-1], "key") else str(path[-1])
if name == "kernel":
- return _fake_quantize_int4(leaf)
+ return _fake_quantize_int4(leaf, group_size=group_size)
return leaf
return jax.tree_util.tree_map_with_path(_maybe_quantize, params)
+# Module-level group_size used by _train_step (set before pmap in train())
+_GROUP_SIZE = 32
+
+
def _train_step(state, ema_params, src, tgt_in, tgt_out, causal_mask, dropout_rng):
pad_id = 0
ema_decay = 0.999
@@ -224,7 +297,7 @@ def _train_step(state, ema_params, src, tgt_in, tgt_out, causal_mask, dropout_rn
cross_mask = src_mask
logits = state.apply_fn(
- {"params": _quantize_params(params)},
+ {"params": _quantize_params(params, group_size=_GROUP_SIZE)},
src,
tgt_in,
src_mask=src_mask,
@@ -250,7 +323,8 @@ def _train_step(state, ema_params, src, tgt_in, tgt_out, causal_mask, dropout_rn
return state, ema_params, loss, grad_norm
-p_train_step = jax.pmap(_train_step, axis_name="batch", donate_argnums=(0, 1))
+def _make_p_train_step():
+ return jax.pmap(_train_step, axis_name="batch", donate_argnums=(0, 1))
def _make_val_loss_fn(apply_fn):
@@ -320,6 +394,11 @@ def train(args):
dtype=args.dtype,
)
+ # Set group size for QAT and sparsification (captured by _train_step closure)
+ global _GROUP_SIZE
+ _GROUP_SIZE = getattr(args, "group_size", 32)
+ p_train_step = _make_p_train_step()
+
np.random.seed(args.seed)
rng = jax.random.PRNGKey(args.seed)
rng, init_rng = jax.random.split(rng)
@@ -361,21 +440,28 @@ def train(args):
dec_targets = jnp.array(dec_targets)
last_val_ppl = None
- prune_ratio = getattr(args, "prune_ratio", 0.0)
+ sparsity_ratio = getattr(args, "sparsity_ratio", 0.0)
+ layer_prune_ratio = getattr(args, "layer_prune_ratio", 0.0)
prune_mask = None
+ # Determine which epoch each pruning type fires on:
+ # both flags: group sparsify @ epoch 1, layer prune @ epoch 2
+ # only --sparsity-ratio: group sparsify @ epoch 1
+ # only --layer-prune: layer prune @ epoch 1
+ weight_prune_epoch = 1 if sparsity_ratio > 0 else -1
+ layer_prune_epoch = (2 if sparsity_ratio > 0 else 1) if layer_prune_ratio > 0 else -1
+
for epoch in range(args.epochs):
- # After first epoch, prune and lock the mask for remaining epochs
- if epoch == 1 and prune_ratio > 0 and prune_mask is None:
- print(f"\nPruning {prune_ratio*100:.0f}% of weight blocks (group_size=32)...")
- pruned_params = _block_prune_params(jax_utils.unreplicate(ema_params), prune_ratio=prune_ratio)
+ # --- Weight block pruning ---
+ if epoch == weight_prune_epoch and prune_mask is None:
+ print(f"\nGroup-wise sparsification: {sparsity_ratio*100:.0f}% of blocks (group_size={_GROUP_SIZE})...")
+ pruned_params = _block_prune_params(jax_utils.unreplicate(ema_params), prune_ratio=sparsity_ratio, group_size=_GROUP_SIZE)
pruned_np = jax.tree.map(np.array, pruned_params)
total_p = sum(x.size for x in jax.tree.leaves(pruned_np))
zero_p = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(pruned_np))
print(f" Post-prune sparsity: {zero_p/total_p*100:.2f}% ({zero_p:,}/{total_p:,} near-zero)")
- # Eval val perplexity after pruning
val_causal = make_causal_mask(args.max_dec_len)
total_loss, total_toks = 0.0, 0.0
for vb in get_batches(val_enc, val_dec_in, val_dec_tgt, args.batch_size, shuffle=False):
@@ -385,7 +471,6 @@ def train(args):
prune_val_ppl = float(math.exp(total_loss / max(total_toks, 1)))
print(f" Post-prune val ppl: {prune_val_ppl:.2f} (was {last_val_ppl:.2f})" if last_val_ppl is not None else f" Post-prune val ppl: {prune_val_ppl:.2f}")
- # Inject pruned params and build frozen mask
state = jax_utils.unreplicate(state)
state = state.replace(params=pruned_params)
ema_params = jax.tree.map(jnp.copy, pruned_params)
@@ -393,7 +478,43 @@ def train(args):
prune_mask = jax_utils.replicate(prune_mask)
state = jax_utils.replicate(state)
ema_params = jax_utils.replicate(ema_params)
- print(f"Continuing training with pruning mask locked...\n")
+ print(f"Continuing training with sparsity mask locked...\n")
+
+ # --- Layer pruning ---
+ if epoch == layer_prune_epoch:
+ print(f"\nLayer pruning: removing {layer_prune_ratio*100:.0f}% of layers...")
+ current_params = jax_utils.unreplicate(ema_params)
+ new_params, config = _layer_prune(current_params, config, layer_prune_ratio=layer_prune_ratio)
+
+ # Rebuild train state with new config/model so tree types are consistent
+ rng, reinit_rng = jax.random.split(rng)
+ state = create_train_state(reinit_rng, config, scaled_lr, muon_lr, total_steps, warmup_steps)
+ matched_params = jax.tree.map(lambda init_leaf, prune_leaf: prune_leaf, state.params, new_params)
+ state = state.replace(params=matched_params)
+ ema_params = jax.tree.map(jnp.copy, matched_params)
+
+ param_count = sum(x.size for x in jax.tree.leaves(matched_params))
+ print(f" Parameters: {param_count:,}")
+
+ # Eval val perplexity after layer pruning
+ val_loss_fn = _make_val_loss_fn(state.apply_fn)
+ val_causal = make_causal_mask(args.max_dec_len)
+ total_loss, total_toks = 0.0, 0.0
+ for vb in get_batches(val_enc, val_dec_in, val_dec_tgt, args.batch_size, shuffle=False):
+ vl, vt = val_loss_fn(matched_params, vb[0], vb[1], vb[2], val_causal)
+ total_loss += float(vl)
+ total_toks += float(vt)
+ layer_prune_ppl = float(math.exp(total_loss / max(total_toks, 1)))
+ print(f" Post-layer-prune val ppl: {layer_prune_ppl:.2f} (was {last_val_ppl:.2f})" if last_val_ppl is not None else f" Post-layer-prune val ppl: {layer_prune_ppl:.2f}")
+
+ # Rebuild prune mask if weight pruning was already applied
+ if prune_mask is not None:
+ prune_mask = jax.tree.map(lambda w: (jnp.abs(w) > 1e-8).astype(jnp.float32), matched_params)
+ prune_mask = jax_utils.replicate(prune_mask)
+
+ state = jax_utils.replicate(state)
+ ema_params = jax_utils.replicate(ema_params)
+ print(f"Continuing training with {config.num_encoder_layers}+{config.num_decoder_layers} layers...\n")
losses = []
batches = get_batches(enc_inputs, dec_inputs, dec_targets, effective_batch_size)