Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
pruning techniques
HenryNdubuaku3940814ea1a1
diff --git a/experiements/pruning.txt b/experiements/pruning.txtnew file mode 100644index 0000000..2a5f8e9--- /dev/null+++ b/experiements/pruning.txt@@ -0,0 +1,388 @@+================================================================================+STRUCTURED COMPRESSION OF SMALL ENCODER-DECODER TRANSFORMERS:+JOINT GROUP-WISE SPARSIFICATION AND LAYER PRUNING+================================================================================+Target venue: ICML 2026++Thesis: Structured compression (group-aligned block sparsity + layer removal)+applied in a staged schedule during training yields Pareto-optimal accuracy-+efficiency tradeoffs for small encoder-decoder models, matching or exceeding+post-hoc pruning at equivalent FLOPs reduction while preserving INT4+quantization compatibility.++================================================================================+SECTION 1 — EXPERIMENTAL AXES+================================================================================++Independent variables:+ A. Sparsity ratio s ∈ {0.0, 0.10, 0.25, 0.40, 0.50}+ B. Layer prune ratio l ∈ {0.0, 0.125, 0.25, 0.375, 0.50}+ C. Group size g ∈ {16, 32, 64, 128}+ D. Pruning schedule ∈ {staged, simultaneous, reverse-order}+ E. Model scale ∈ {BASE-256, LARGE-512}+ F. QAT ∈ {off, int4-group32}++Dependent variables (measured):+ - Validation perplexity (TinyStories)+ - WikiText-2 perplexity+ - LAMBADA accuracy+ - HellaSwag accuracy+ - ARC-Easy accuracy+ - Throughput (tokens/sec on inference)+ - Bigram repetition rate (generation diversity)+ - Parameter count (absolute and % of dense baseline)+ - Effective FLOPs (params * density)+ - Weight sparsity (% near-zero, measured at threshold < 1e-6)++================================================================================+SECTION 2 — MODEL CONFIGURATIONS+================================================================================++BASE-256 (25.4M params, default):+ d_model=256, num_heads=4, num_kv_heads=2, num_layers=4,+ d_ff=1024, max_seq_len=256, vocab=65536, dtype=bfloat16++LARGE-512 (≈100M params, scaling study):+ d_model=512, num_heads=8, num_kv_heads=4, num_layers=8,+ d_ff=2048, max_seq_len=256, vocab=65536, dtype=bfloat16++Both trained on TinyStories (500K samples, encoder-decoder split at 30%).+Optimizer: Muon (kernels) + AdamW (embeddings/norms), WSD schedule.++================================================================================+SECTION 3 — EXPERIMENT GROUPS+================================================================================++All experiments use 3 seeds (42, 137, 256) for statistical significance.+Report mean ± std on all metrics.++-------------------------------------------------------------------------------+GROUP 1: Dense Baselines (Table 1)+-------------------------------------------------------------------------------+Purpose: Establish upper-bound quality for each model scale.++ 1a. BASE-256, 3 epochs, no pruning, no QAT+ needle train --epochs 3 --seed {42,137,256}++ 1b. BASE-256, 3 epochs, no pruning, with QAT (int4-group32)+ [QAT is always on in current code — this IS the baseline]++ 1c. LARGE-512, 3 epochs, no pruning+ needle train --epochs 3 --d-model 512 --num-heads 8 --num-layers 8 \+ --max-enc-len 256 --max-dec-len 256 --seed {42,137,256}++Deliverables:+ - Val perplexity, benchmark scores, throughput, generation samples+ - These are the denominators for all relative comparisons++-------------------------------------------------------------------------------+GROUP 2: Group-Wise Sparsification Only (Table 2, Figure 1)+-------------------------------------------------------------------------------+Purpose: Characterise accuracy-sparsity Pareto frontier for block pruning alone.+Schedule: Epoch 1 dense, epochs 2-3 with sparsity mask locked.++ For s in {0.10, 0.25, 0.40, 0.50}:+ 2a. BASE-256, 3 epochs, --sparsity-ratio s+ needle train --epochs 3 --sparsity-ratio {s} --seed {42,137,256}++ For s in {0.10, 0.25, 0.40, 0.50}:+ 2b. LARGE-512, 3 epochs, --sparsity-ratio s+ needle train --epochs 3 --d-model 512 --num-heads 8 --num-layers 8 \+ --sparsity-ratio {s} --seed {42,137,256}++Deliverables:+ - Pareto curve: sparsity % vs val perplexity (Figure 1a)+ - Pareto curve: sparsity % vs LAMBADA accuracy (Figure 1b)+ - Table: all metrics at each sparsity level+ - Analysis: at what sparsity does quality degrade >5% relative to baseline?++-------------------------------------------------------------------------------+GROUP 3: Layer Pruning Only (Table 3, Figure 2)+-------------------------------------------------------------------------------+Purpose: Characterise accuracy-compression frontier for layer removal alone.+Schedule: Epoch 1 dense, epochs 2-3 with reduced architecture.++ For l in {0.125, 0.25, 0.375, 0.50}:+ 3a. BASE-256 (4+4 layers), 3 epochs, --layer-prune-ratio l+ needle train --epochs 3 --layer-prune-ratio {l} --seed {42,137,256}+ Expected surviving layers: 7, 6, 5, 4 (of 8 total)++ For l in {0.125, 0.25, 0.375, 0.50}:+ 3b. LARGE-512 (8+8 layers), 3 epochs, --layer-prune-ratio l+ needle train --epochs 3 --d-model 512 --num-heads 8 --num-layers 8 \+ --layer-prune-ratio {l} --seed {42,137,256}+ Expected surviving layers: 14, 12, 10, 8 (of 16 total)++Deliverables:+ - Pareto curve: layer ratio vs val perplexity (Figure 2a)+ - Pareto curve: parameter count vs LAMBADA accuracy (Figure 2b)+ - Table: per-run encoder/decoder surviving layer counts+ - Analysis: does the pruner prefer removing encoder or decoder layers?+ (Hypothesis: encoder layers are pruned more aggressively since the+ decoder carries more of the generation burden.)++-------------------------------------------------------------------------------+GROUP 4: Joint Sparsification + Layer Pruning (Table 4, Figure 3)+ *** CORE CONTRIBUTION ***+-------------------------------------------------------------------------------+Purpose: Show that combining both methods outperforms either alone at matched+compression ratio. This is the central claim of the paper.++Schedule (staged):+ Epoch 1: dense training+ Epoch 2: apply group-wise sparsification, train with mask+ Epoch 3: apply layer pruning, train with reduced architecture + mask+ Epoch 4: recovery fine-tuning++ For (s, l) in grid:+ {(0.10, 0.125), (0.10, 0.25),+ (0.25, 0.125), (0.25, 0.25), (0.25, 0.375),+ (0.40, 0.125), (0.40, 0.25),+ (0.50, 0.25), (0.50, 0.50)}++ 4a. BASE-256, 4 epochs+ needle train --epochs 4 --sparsity-ratio {s} --layer-prune-ratio {l} \+ --seed {42,137,256}++ 4b. LARGE-512, select best 4 (s,l) pairs from 4a, 4 epochs+ needle train --epochs 4 --d-model 512 --num-heads 8 --num-layers 8 \+ --sparsity-ratio {s} --layer-prune-ratio {l} --seed {42,137,256}++Deliverables:+ - 2D heatmap: (s, l) → val perplexity (Figure 3a)+ - 2D heatmap: (s, l) → effective parameter count (Figure 3b)+ - Pareto overlay: joint vs sparsity-only vs layer-only at matched+ compression (Figure 3c) — THIS IS THE KEY FIGURE+ - Table: full metrics for all (s, l) pairs+ - Analysis: multiplicative vs additive compression benefit++-------------------------------------------------------------------------------+GROUP 5: Schedule Ablation (Table 5)+-------------------------------------------------------------------------------+Purpose: Justify the staged schedule (sparsify first, then prune layers).++Pick the best (s, l) pair from Group 4a. Run 3 schedules:++ 5a. Staged (default): sparsify epoch 2, layer prune epoch 3, recover epoch 4+ needle train --epochs 4 --sparsity-ratio {s} --layer-prune-ratio {l}++ 5b. Reverse: layer prune epoch 2, sparsify epoch 3, recover epoch 4+ [Requires code change to swap epoch triggers]++ 5c. Simultaneous: both at epoch 2, recover epochs 3-4+ [Requires code change to apply both at same epoch]++Deliverables:+ - Table: val perplexity and benchmark scores for each schedule+ - Analysis: why staged order matters (hypothesis: sparsification is+ lower-risk, lets model adapt before the more destructive layer removal)++-------------------------------------------------------------------------------+GROUP 6: Recovery Epoch Scaling (Figure 4)+-------------------------------------------------------------------------------+Purpose: How many recovery epochs are needed after compression?++Pick the best (s, l) pair from Group 4a. Vary total epochs:++ For E in {2, 3, 4, 5, 6}:+ 6a. BASE-256, E epochs, --sparsity-ratio {s} --layer-prune-ratio {l}+ (pruning always at epochs 2 and 3; recovery = E - 3 epochs)++Deliverables:+ - Line plot: recovery epochs vs val perplexity (Figure 4)+ - Analysis: diminishing returns curve, recommend minimum recovery budget++-------------------------------------------------------------------------------+GROUP 7: QAT Interaction Study (Table 6)+-------------------------------------------------------------------------------+Purpose: Show that group-wise sparsification is compatible with INT4 QAT+because both operate on the same group_size=32 granularity.++ 7a. Dense + QAT (baseline, = Group 1b)+ 7b. Sparsified (s=0.25) + QAT+ 7c. Layer-pruned (l=0.25) + QAT+ 7d. Joint (s=0.25, l=0.25) + QAT++ For comparison, also run 7b-7d WITHOUT QAT by modifying _train_step+ to not call _quantize_params.++Deliverables:+ - Table: QAT on vs off for each compression method+ - Analysis: does QAT hurt more when combined with sparsification?+ (Hypothesis: no, because zeroed groups have no quantization error)+ - Effective compression ratio: sparsity + INT4 → bits per parameter++-------------------------------------------------------------------------------+GROUP 8: Group Size Ablation (Table 7, Figure 6)+-------------------------------------------------------------------------------+Purpose: Determine optimal granularity for block sparsification. Smaller groups+give finer-grained pruning decisions (better accuracy retention) but produce+less structured sparsity (harder to exploit in hardware). Larger groups are+more hardware-friendly but coarser. The group size also controls QAT+quantization granularity, so this directly tests the sparsity-quantization+co-design hypothesis.++Note: requires adding --group-size CLI flag and threading it through to+_block_prune_params and _fake_quantize_int4.++Fixed configuration: BASE-256, s=0.25, 3 epochs, QAT on.++ For g in {16, 32, 64, 128}:+ 8a. Sparsification only (isolate group size effect on pruning)+ needle train --epochs 3 --sparsity-ratio 0.25 --group-size {g} \+ --seed {42,137,256}++ 8b. Joint sparsification + layer pruning (best l from Group 4a)+ needle train --epochs 4 --sparsity-ratio 0.25 --layer-prune-ratio {l} \+ --group-size {g} --seed {42,137,256}++Additional comparison at high sparsity to stress-test granularity:++ For g in {16, 32, 64, 128}:+ 8c. s=0.50, sparsification only+ needle train --epochs 3 --sparsity-ratio 0.50 --group-size {g} \+ --seed {42,137,256}++Deliverables:+ - Line plot: group size vs val perplexity at s=0.25 and s=0.50 (Figure 6a)+ - Line plot: group size vs LAMBADA accuracy (Figure 6b)+ - Table: all metrics at each group size for both sparsity levels+ - Analysis of key tradeoffs:+ (a) g=16: finest granularity, best accuracy, but 2× more scale factors+ to store than g=32 and least structured pattern+ (b) g=32: default, matches common INT4 quantization group size,+ good balance of accuracy and structure+ (c) g=64: coarser, each pruning decision covers 64 weights,+ more aggressive but more hardware-friendly+ (d) g=128: coarsest, each block is a full row for d_model=128,+ approaches per-channel granularity, may degrade significantly+ - Cross-reference with QAT: when group_size is shared between sparsification+ and quantization, zeroed groups save both weight storage AND scale storage+ (no scale needed for an all-zero group). Report effective bits-per-param+ accounting for this: bpp = (1 - sparsity) × (4 + 32/g) where 4 is INT4+ bits and 32/g is amortized scale overhead per weight.++-------------------------------------------------------------------------------+GROUP 9: Inference Efficiency Validation (Table 8, Figure 7)+-------------------------------------------------------------------------------+Purpose: Translate theoretical compression into real speedup.++For each Pareto-optimal checkpoint from Groups 2-4:+ needle test --checkpoint {path} --throughput-runs 50++Measure:+ - Tokens/sec (decode throughput)+ - Latency per generation (ms)+ - Peak memory (via JAX profiler)++Deliverables:+ - Scatter plot: compression ratio vs actual speedup (Figure 5)+ - Table: all Pareto-optimal models with throughput + quality+ - Analysis: does structured sparsity translate to real speedup?+ (Note: dense JAX matmuls don't skip zeros — speedup comes from+ layer removal. Sparsity benefit is in model size / memory only+ until sparse kernel support is added.)++================================================================================+SECTION 4 — EVALUATION PROTOCOL+================================================================================++For every checkpoint:++ 1. Validation perplexity (TinyStories val split, full)+ needle test --checkpoint {path} --max-eval-samples 5000++ 2. Benchmark suite+ needle evaluate --checkpoint {path} \+ --benchmarks wikitext2 lambada hellaswag arc_easy --max-samples 500++ 3. Generation quality (qualitative + bigram diversity)+ needle test --checkpoint {path}+ → Inspect 10 sample generations, report bigram repetition rate++ 4. Throughput measurement+ needle test --checkpoint {path} --throughput-runs 50++================================================================================+SECTION 5 — EXPECTED TABLES AND FIGURES+================================================================================++Tables:+ T1. Dense baselines (2 scales × {with,without} QAT)+ T2. Group-wise sparsification sweep (2 scales × 4 ratios)+ T3. Layer pruning sweep (2 scales × 4 ratios)+ T4. Joint compression grid (9 pairs × 2 scales)+ T5. Schedule ablation (3 schedules)+ T6. QAT interaction (4 methods × {on,off})+ T7. Group size ablation (4 sizes × 2 sparsity levels + joint)+ T8. Inference efficiency (Pareto-optimal checkpoints)++Figures:+ F1. Sparsity-only Pareto: sparsity % vs perplexity + LAMBADA+ F2. Layer-prune-only Pareto: param count vs perplexity + LAMBADA+ F3. Joint compression: (a) heatmap, (b) param heatmap, (c) Pareto overlay+ F4. Recovery epoch scaling curve+ F5. Compression ratio vs actual inference speedup+ F6. Group size: (a) g vs perplexity at s=0.25/0.50, (b) g vs LAMBADA+ F7. Effective bits-per-param vs quality across group sizes++================================================================================+SECTION 6 — COMPUTE BUDGET+================================================================================++Per-run cost (BASE-256, 4 epochs, 500K samples, 4× TPU v5 lite):+ ~10 minutes training + ~2 minutes evaluation++Estimated total runs:+ Group 1: 2 configs × 3 seeds = 6 runs+ Group 2: 4 ratios × 2 scales × 3 seeds = 24 runs+ Group 3: 4 ratios × 2 scales × 3 seeds = 24 runs+ Group 4: 9 pairs × 3 seeds + 4 pairs × 3 seeds (large) = 39 runs+ Group 5: 3 schedules × 3 seeds = 9 runs+ Group 6: 5 epoch counts × 3 seeds = 15 runs+ Group 7: 4 methods × 2 (QAT on/off) × 3 seeds = 24 runs+ Group 8: 4 sizes × 3 configs (s=.25, s=.50, joint) × 3 = 36 runs+ Group 9: evaluation only (reuses checkpoints) = 0 runs+ ─────────────────────────────────────────────────────────────────────+ Total: 177 runs++At ~12 min/run: ~35 TPU-hours (very manageable on v5litepod-4)++================================================================================+SECTION 7 — ABLATION CHECKLIST+================================================================================++Before submission, verify these are addressed:++ [ ] Statistical significance: 3 seeds, report mean ± std+ [ ] Matched comparisons: compare joint vs individual at SAME compression+ [ ] Scaling behavior: BASE-256 and LARGE-512 show consistent trends+ [ ] Schedule sensitivity: staged vs reverse vs simultaneous+ [ ] Recovery saturation: diminishing returns on fine-tuning epochs+ [ ] QAT compatibility: sparsity + quantization are not antagonistic+ [ ] Pruning bias: report which layers (enc vs dec) get pruned+ [ ] Generation quality: not just perplexity, show actual text samples+ [ ] Group size sensitivity: optimal g, and does it interact with model scale?+ [ ] Honest limitations: sparse matmuls don't accelerate on current JAX++================================================================================+SECTION 8 — EXECUTION ORDER+================================================================================++Phase 1 (Day 1): Groups 1 + 2 + 3 (54 runs, ~11 hrs)+ Run all baselines and single-method sweeps in parallel across seeds.+ Immediate deliverable: Pareto curves for each method alone.++Phase 2 (Day 2): Group 4 (39 runs, ~8 hrs)+ Joint compression grid. This produces the key result.+ Immediate deliverable: heatmaps and overlay Pareto.++Phase 3 (Day 3): Groups 5 + 6 + 7 + 8 (84 runs, ~17 hrs)+ Ablations: schedule, recovery, QAT interaction, group size.+ Immediate deliverable: complete ablation tables.++Phase 4 (Day 4): Group 9 + writing+ Run inference benchmarks on all Pareto-optimal checkpoints.+ Begin paper draft with all tables and figures populated.++================================================================================diff --git a/src/cli.py b/src/cli.pyindex d55465d..51e8446 100644--- a/src/cli.py+++ b/src/cli.py@@ -137,7 +137,9 @@ def main():p.add_argument("--seed", type=int, default=None)p.add_argument("--eval-every", type=int, default=1000)p.add_argument("--max-eval-samples", type=int, default=None)- p.add_argument("--prune-ratio", type=float, default=0.0)+ p.add_argument("--sparsity-ratio", type=float, default=0.33)+ p.add_argument("--layer-prune-ratio", type=float, default=0.33)+ p.add_argument("--group-size", type=int, default=32)p = sub.add_parser("run", add_help=False)p.add_argument("--checkpoint", type=str, required=True)diff --git a/src/train.py b/src/train.pyindex 35b2006..8f65ee9 100644--- a/src/train.py+++ b/src/train.py@@ -204,16 +204,89 @@ def _block_prune_params(params, prune_ratio=0.25, group_size=32):return jax.tree_util.tree_map_with_path(_prune_weight, params)-def _quantize_params(params):+def _layer_prune(params, config, layer_prune_ratio=0.25):+ """Remove the lowest-scoring encoder/decoder blocks by L1 magnitude.++ Returns (new_params, new_config) with surviving blocks renumbered.+ """+ from flax.core import unfreeze+ p = unfreeze(params)++ def _score_blocks(module_params, prefix):+ scores = {}+ for key, val in module_params.items():+ if key.startswith("block_"):+ s = sum(float(np.sum(np.abs(np.array(l)))) for l in jax.tree.leaves(val))+ scores[key] = s+ return scores++ enc_scores = _score_blocks(p["encoder"], "encoder")+ dec_scores = _score_blocks(p["decoder"], "decoder")++ # Pool all block scores, find global threshold+ all_scores = list(enc_scores.values()) + list(dec_scores.values())+ if not all_scores:+ return params, config++ # Keep at least 1 encoder and 1 decoder block+ num_enc = len(enc_scores)+ num_dec = len(dec_scores)+ total_to_remove = max(0, int(len(all_scores) * layer_prune_ratio))++ # Score and rank all blocks together+ tagged = [(s, "encoder", k) for k, s in enc_scores.items()] + \+ [(s, "decoder", k) for k, s in dec_scores.items()]+ tagged.sort(key=lambda x: x[0])++ remove_enc = set()+ remove_dec = set()+ for score, module, key in tagged:+ if len(remove_enc) + len(remove_dec) >= total_to_remove:+ break+ if module == "encoder" and (num_enc - len(remove_enc)) > 1:+ remove_enc.add(key)+ elif module == "decoder" and (num_dec - len(remove_dec)) > 1:+ remove_dec.add(key)++ # Rebuild with surviving blocks renumbered+ def _keep_and_renumber(module_params, remove_set):+ surviving = [(k, v) for k, v in sorted(module_params.items()) if k.startswith("block_") and k not in remove_set]+ non_blocks = {k: v for k, v in module_params.items() if not k.startswith("block_")}+ new = dict(non_blocks)+ for i, (_, v) in enumerate(surviving):+ new[f"block_{i}"] = v+ return new++ p["encoder"] = _keep_and_renumber(p["encoder"], remove_enc)+ p["decoder"] = _keep_and_renumber(p["decoder"], remove_dec)++ new_num_enc = num_enc - len(remove_enc)+ new_num_dec = num_dec - len(remove_dec)+ print(f" Encoder: {num_enc} -> {new_num_enc} blocks (removed {len(remove_enc)})")+ print(f" Decoder: {num_dec} -> {new_num_dec} blocks (removed {len(remove_dec)})")++ from dataclasses import replace as dc_replace+ new_config = dc_replace(config,+ num_encoder_layers=new_num_enc,+ num_decoder_layers=new_num_dec)++ return p, new_config+++def _quantize_params(params, group_size=32):"""Fake-quantize all Dense kernels in the param tree."""def _maybe_quantize(path, leaf):name = path[-1].key if hasattr(path[-1], "key") else str(path[-1])if name == "kernel":- return _fake_quantize_int4(leaf)+ return _fake_quantize_int4(leaf, group_size=group_size)return leafreturn jax.tree_util.tree_map_with_path(_maybe_quantize, params)+# Module-level group_size used by _train_step (set before pmap in train())+_GROUP_SIZE = 32++def _train_step(state, ema_params, src, tgt_in, tgt_out, causal_mask, dropout_rng):pad_id = 0ema_decay = 0.999@@ -224,7 +297,7 @@ def _train_step(state, ema_params, src, tgt_in, tgt_out, causal_mask, dropout_rncross_mask = src_masklogits = state.apply_fn(- {"params": _quantize_params(params)},+ {"params": _quantize_params(params, group_size=_GROUP_SIZE)},src,tgt_in,src_mask=src_mask,@@ -250,7 +323,8 @@ def _train_step(state, ema_params, src, tgt_in, tgt_out, causal_mask, dropout_rnreturn state, ema_params, loss, grad_norm-p_train_step = jax.pmap(_train_step, axis_name="batch", donate_argnums=(0, 1))+def _make_p_train_step():+ return jax.pmap(_train_step, axis_name="batch", donate_argnums=(0, 1))def _make_val_loss_fn(apply_fn):@@ -320,6 +394,11 @@ def train(args):dtype=args.dtype,)+ # Set group size for QAT and sparsification (captured by _train_step closure)+ global _GROUP_SIZE+ _GROUP_SIZE = getattr(args, "group_size", 32)+ p_train_step = _make_p_train_step()+np.random.seed(args.seed)rng = jax.random.PRNGKey(args.seed)rng, init_rng = jax.random.split(rng)@@ -361,21 +440,28 @@ def train(args):dec_targets = jnp.array(dec_targets)last_val_ppl = None- prune_ratio = getattr(args, "prune_ratio", 0.0)+ sparsity_ratio = getattr(args, "sparsity_ratio", 0.0)+ layer_prune_ratio = getattr(args, "layer_prune_ratio", 0.0)prune_mask = None+ # Determine which epoch each pruning type fires on:+ # both flags: group sparsify @ epoch 1, layer prune @ epoch 2+ # only --sparsity-ratio: group sparsify @ epoch 1+ # only --layer-prune: layer prune @ epoch 1+ weight_prune_epoch = 1 if sparsity_ratio > 0 else -1+ layer_prune_epoch = (2 if sparsity_ratio > 0 else 1) if layer_prune_ratio > 0 else -1+for epoch in range(args.epochs):- # After first epoch, prune and lock the mask for remaining epochs- if epoch == 1 and prune_ratio > 0 and prune_mask is None:- print(f"\nPruning {prune_ratio*100:.0f}% of weight blocks (group_size=32)...")- pruned_params = _block_prune_params(jax_utils.unreplicate(ema_params), prune_ratio=prune_ratio)+ # --- Weight block pruning ---+ if epoch == weight_prune_epoch and prune_mask is None:+ print(f"\nGroup-wise sparsification: {sparsity_ratio*100:.0f}% of blocks (group_size={_GROUP_SIZE})...")+ pruned_params = _block_prune_params(jax_utils.unreplicate(ema_params), prune_ratio=sparsity_ratio, group_size=_GROUP_SIZE)pruned_np = jax.tree.map(np.array, pruned_params)total_p = sum(x.size for x in jax.tree.leaves(pruned_np))zero_p = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(pruned_np))print(f" Post-prune sparsity: {zero_p/total_p*100:.2f}% ({zero_p:,}/{total_p:,} near-zero)")- # Eval val perplexity after pruningval_causal = make_causal_mask(args.max_dec_len)total_loss, total_toks = 0.0, 0.0for vb in get_batches(val_enc, val_dec_in, val_dec_tgt, args.batch_size, shuffle=False):@@ -385,7 +471,6 @@ def train(args):prune_val_ppl = float(math.exp(total_loss / max(total_toks, 1)))print(f" Post-prune val ppl: {prune_val_ppl:.2f} (was {last_val_ppl:.2f})" if last_val_ppl is not None else f" Post-prune val ppl: {prune_val_ppl:.2f}")- # Inject pruned params and build frozen maskstate = jax_utils.unreplicate(state)state = state.replace(params=pruned_params)ema_params = jax.tree.map(jnp.copy, pruned_params)@@ -393,7 +478,43 @@ def train(args):prune_mask = jax_utils.replicate(prune_mask)state = jax_utils.replicate(state)ema_params = jax_utils.replicate(ema_params)- print(f"Continuing training with pruning mask locked...\n")+ print(f"Continuing training with sparsity mask locked...\n")++ # --- Layer pruning ---+ if epoch == layer_prune_epoch:+ print(f"\nLayer pruning: removing {layer_prune_ratio*100:.0f}% of layers...")+ current_params = jax_utils.unreplicate(ema_params)+ new_params, config = _layer_prune(current_params, config, layer_prune_ratio=layer_prune_ratio)++ # Rebuild train state with new config/model so tree types are consistent+ rng, reinit_rng = jax.random.split(rng)+ state = create_train_state(reinit_rng, config, scaled_lr, muon_lr, total_steps, warmup_steps)+ matched_params = jax.tree.map(lambda init_leaf, prune_leaf: prune_leaf, state.params, new_params)+ state = state.replace(params=matched_params)+ ema_params = jax.tree.map(jnp.copy, matched_params)++ param_count = sum(x.size for x in jax.tree.leaves(matched_params))+ print(f" Parameters: {param_count:,}")++ # Eval val perplexity after layer pruning+ val_loss_fn = _make_val_loss_fn(state.apply_fn)+ val_causal = make_causal_mask(args.max_dec_len)+ total_loss, total_toks = 0.0, 0.0+ for vb in get_batches(val_enc, val_dec_in, val_dec_tgt, args.batch_size, shuffle=False):+ vl, vt = val_loss_fn(matched_params, vb[0], vb[1], vb[2], val_causal)+ total_loss += float(vl)+ total_toks += float(vt)+ layer_prune_ppl = float(math.exp(total_loss / max(total_toks, 1)))+ print(f" Post-layer-prune val ppl: {layer_prune_ppl:.2f} (was {last_val_ppl:.2f})" if last_val_ppl is not None else f" Post-layer-prune val ppl: {layer_prune_ppl:.2f}")++ # Rebuild prune mask if weight pruning was already applied+ if prune_mask is not None:+ prune_mask = jax.tree.map(lambda w: (jnp.abs(w) > 1e-8).astype(jnp.float32), matched_params)+ prune_mask = jax_utils.replicate(prune_mask)++ state = jax_utils.replicate(state)+ ema_params = jax_utils.replicate(ema_params)+ print(f"Continuing training with {config.num_encoder_layers}+{config.num_decoder_layers} layers...\n")losses = []batches = get_batches(enc_inputs, dec_inputs, dec_targets, effective_batch_size)