Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

Add experiments

HenryNdubuaku126117fafdd2
diff --git a/README.md b/README.md
index 1a124f6..ca53428 100644
--- a/README.md
+++ b/README.md
@@ -160,23 +160,23 @@ needle [command]
~27 GB (from ~4.5 TB full dataset)
──────────────────────────────────────────
- 300M multimodal training estimates
+ 1B multimodal training estimates
──────────────────────────────────────────
- Total FLOPs 6 × 300M × 100B = 1.8e20
- MFU (300M model) ~15% → ~138 effective TFLOPS/chip
+ Total FLOPs 6 × 1B × 100B = 6e20
+ MFU (1B model) ~30% → ~275 effective TFLOPS/chip
Audio overhead <1% of total (negligible)
- ┌────────────────────┬──────────┬──────────┐
- │ │ v6e-8 │ v6e-16 │
- ├────────────────────┼──────────┼──────────┤
- │ Chips │ 8 │ 16 │
- │ Total HBM │ 256 GB │ 512 GB │
- │ Scaling eff. │ 0.9× │ 0.8× │
- │ Eff. TFLOPS │ 992 │ 1,766 │
- │ Est. time │ ~50h │ ~29h │
- │ On-demand $/hr │ $21.60 │ $43.20 │
- │ Est. total cost │ ~$1,080 │ ~$1,250 │
- └────────────────────┴──────────┴──────────┘
+ ┌────────────────────┬──────────┬──────────┬──────────┐
+ │ │ v6e-8 │ v6e-16 │ v6e-32 │
+ ├────────────────────┼──────────┼──────────┼──────────┤
+ │ Chips │ 8 │ 16 │ 32 │
+ │ Total HBM │ 256 GB │ 512 GB │ 1024 GB │
+ │ Scaling eff. │ 0.9× │ 0.8× │ 0.7× │
+ │ Eff. TFLOPS │ 1,983 │ 3,525 │ 6,160 │
+ │ Est. time │ ~84h │ ~47h │ ~27h │
+ │ On-demand $/hr │ $21.60 │ $43.20 │ $86.40 │
+ │ Est. total cost │ ~$1,810 │ ~$2,030 │ ~$2,330 │
+ └────────────────────┴──────────┴──────────┴──────────┘
```
## Setup For TPU/GCP
diff --git a/experiements/architecture.txt b/experiements/architecture.txt
index e69de29..50965f5 100644
--- a/experiements/architecture.txt
+++ b/experiements/architecture.txt
@@ -0,0 +1,390 @@
+================================================================================
+Assignee: Noah Cylich & Justin Lee
+Deadline: 7th March
+================================================================================
+
+Thesis: Three complementary compression axes; Matryoshka dimension pruning,
+group-aligned block sparsity, and INT4 quantization-aware training. can be
+jointly applied during training to produce a single checkpoint that is
+deployable at multiple accuracy-efficiency operating points. Training once at
+d_model=D with MRL loss at dimensions {D, D/2, D/4, D/8, ...} yields a family
+of models from one training run, each further compressible via sparsification.
+
+================================================================================
+SECTION 1 — EXPERIMENTAL AXES
+================================================================================
+
+Independent variables:
+ A. MRL deployment dim d' ∈ {1024, 512, 256, 128, 64}
+ B. Sparsity ratio s ∈ {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}
+ C. Group size g ∈ {16, 32, 64, 128}
+ D. FFN activation ∈ {dReLU, SwiGLU, GeGLU}
+ E. Model scale ∈ {TOY-128, MAIN-1024}
+ F. QAT ∈ {off, int4-group32}
+
+Dependent variables (measured at EACH MRL dimension):
+ - Validation perplexity (TinyStories)
+ - Quantized validation perplexity (INT4)
+ - WikiText-2 perplexity
+ - LAMBADA accuracy
+ - HellaSwag accuracy
+ - ARC-Easy accuracy
+ - Estimated parameter count at deployment dim
+ - Throughput (tokens/sec on inference)
+ - Bigram repetition rate (generation diversity)
+ - Block sparsity (% of fully-zero groups, group_size=g)
+ - Effective FLOPs (est. params × density)
+
+================================================================================
+SECTION 2 — MODEL CONFIGURATIONS
+================================================================================
+
+Architecture: MemoryMixer encoder + Transformer decoder, shared embedding,
+GQA (grouped query attention), ZC-RMSNorm, RoPE, SentencePiece BPE (8192).
+
+TOY-128 (rapid iteration):
+ mrl_dims=[128, 64], d_model=128, num_heads=4, num_kv_heads=2,
+ num_enc_layers=2, num_dec_layers=2, d_ff=512,
+ max_enc_len=128, max_dec_len=128, vocab=8192, dtype=bfloat16
+ needle train --toy --seed {42,137,256}
+
+MAIN-1024 (paper results):
+ mrl_dims=[1024, 512, 256, 128, 64], d_model=1024, num_heads=4,
+ num_kv_heads=2, num_enc_layers=4, num_dec_layers=2, d_ff=4096,
+ max_enc_len=256, max_dec_len=256, vocab=8192, dtype=bfloat16
+ needle train --seed {42,137,256}
+
+Both trained on TinyStories (1M samples, encoder-decoder split at 30%).
+Optimizer: Muon (kernels) + AdamW (embeddings/norms), WSD schedule.
+All experiments report metrics at each MRL dimension slice.
+
+================================================================================
+SECTION 3 — EXPERIMENT GROUPS
+================================================================================
+
+All experiments use 3 seeds (42, 137, 256) for statistical significance.
+Report mean +/- std on all metrics. Every run automatically reports val PPL
+and INT4-quantized val PPL at each MRL dimension slice.
+
+-------------------------------------------------------------------------------
+GROUP 1: Dense Baselines + MRL Dimension Scaling (Table 1, Figure 1)
+-------------------------------------------------------------------------------
+Purpose: Establish quality at each deployment dimension from a single training
+run. This is the MRL baseline — no sparsification.
+
+ 1a. MAIN-1024, 3 epochs, no pruning, MRL dims [1024,512,256,128,64]
+ needle train --sparsity-ratio 0.0 --seed {42,137,256}
+
+ 1b. TOY-128, 3 epochs, no pruning, MRL dims [128,64]
+ needle train --toy --sparsity-ratio 0.0 --seed {42,137,256}
+
+Deliverables:
+ - Table: val PPL and quant PPL at each MRL dim (1024, 512, 256, 128, 64)
+ - Line plot: deployment dim vs val PPL — the MRL scaling curve (Figure 1a)
+ - Line plot: deployment dim vs estimated param count (Figure 1b)
+ - Benchmark scores at each dim
+ - These are the denominators for all relative comparisons
+
+-------------------------------------------------------------------------------
+GROUP 2: MRL + Group-Wise Sparsification (Table 2, Figure 2)
+ *** CORE CONTRIBUTION ***
+-------------------------------------------------------------------------------
+Purpose: Characterise how MRL dimension scaling interacts with block sparsity.
+At smaller deployment dims, does sparsification hurt more or less?
+
+Schedule: Epoch 1 dense, epochs 2-3 with sparsity mask locked.
+
+ For s in {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}:
+ 2a. MAIN-1024, 3 epochs, --sparsity-ratio s
+ needle train --sparsity-ratio {s} --seed {42,137,256}
+
+ 2b. Select best 2 sparsity ratios from 2a, run with 4 epochs (recovery)
+ needle train --epochs 4 --sparsity-ratio {s} --seed {42,137,256}
+
+Deliverables:
+ - 2D heatmap: (sparsity, MRL dim) → val PPL (Figure 2a)
+ - Pareto curve per MRL dim: sparsity % vs val PPL (Figure 2b)
+ - Table: all metrics at each (sparsity, dim) pair
+ - Compression spectrum: for best s, show the range of models:
+ d'=1024 (full) → d'=64 (smallest) with estimated params and PPL
+ - Analysis: does sparsification degrade smaller MRL slices more?
+ (Hypothesis: smaller slices are more sensitive because fewer dimensions
+ carry the representation — each zeroed weight matters more)
+ - Analysis: multiplicative compression — MRL dimension reduction ×
+ sparsity = total compression ratio
+
+-------------------------------------------------------------------------------
+GROUP 3: Recovery Epoch Scaling (Table 3, Figure 3)
+-------------------------------------------------------------------------------
+Purpose: How many recovery epochs are needed after sparsification?
+
+Pick the best sparsity ratio from Group 2a. Vary total epochs:
+
+ For E in {2, 3, 4, 5, 6}:
+ 3a. MAIN-1024, E epochs, --sparsity-ratio {s}
+ (sparsification at epoch 1; recovery = E - 1 epochs)
+
+Deliverables:
+ - Line plot per MRL dim: recovery epochs vs val PPL (Figure 3)
+ - Analysis: do smaller MRL dims need more recovery?
+
+-------------------------------------------------------------------------------
+GROUP 4: QAT Interaction Study (Table 4, Figure 4)
+-------------------------------------------------------------------------------
+Purpose: Show that group-wise sparsification is compatible with INT4 QAT
+because both operate on the same group_size=32 granularity. Measure at
+each MRL dimension — INT4 quantization error at smaller dims may compound.
+
+ 4a. Dense + QAT (baseline, = Group 1a)
+ 4b. Sparsified (s=0.3) + QAT
+ 4c. Sparsified (s=0.6) + QAT
+
+ For comparison, also run 4b-4c WITHOUT QAT by modifying _train_step
+ to not call _quantize_params.
+
+Deliverables:
+ - Table: (QAT on/off) × sparsity × MRL dim → val PPL
+ - Analysis: does QAT degrade more at smaller MRL dims?
+ (Hypothesis: yes — fewer dimensions means each quantization error
+ has proportionally larger impact on output quality)
+ - Effective compression: MRL dim reduction × sparsity × INT4 → total
+ bits per effective parameter
+
+-------------------------------------------------------------------------------
+GROUP 5: Group Size Ablation (Table 5, Figure 5)
+-------------------------------------------------------------------------------
+Purpose: Determine optimal block granularity for sparsification.
+
+Fixed: MAIN-1024, s=0.3, 3 epochs, QAT on.
+
+ For g in {16, 32, 64, 128}:
+ 5a. needle train --sparsity-ratio 0.3 --group-size {g} \
+ --seed {42,137,256}
+
+ High sparsity stress test:
+ For g in {16, 32, 64, 128}:
+ 5b. needle train --sparsity-ratio 0.6 --group-size {g} \
+ --seed {42,137,256}
+
+Deliverables:
+ - Line plot per MRL dim: group size vs val PPL (Figure 5a)
+ - Table: all metrics at each (group size, MRL dim)
+ - Effective bits-per-param: bpp = (1 - sparsity) * (4 + 32/g)
+ - Analysis: does optimal group size change with MRL deployment dim?
+
+-------------------------------------------------------------------------------
+GROUP 6: Activation Function x Sparsification Interaction (Table 6, Figure 6)
+ *** KEY FINDING ***
+-------------------------------------------------------------------------------
+Purpose: Test whether activation function choice determines sparsification
+robustness across MRL dimension slices.
+
+Activation definitions:
+ - dReLU: ReLU(gate) * ReLU(up) — naturally ~75% sparse activations
+ - SwiGLU: SiLU(gate) * up — smooth, dense activations
+ - GeGLU: GELU(gate) * up — smooth, nearly dense activations
+
+ 6a. Dense baselines (no sparsification) — isolate activation quality
+ For act in {drelu, swiglu, geglu}:
+ needle train --activation {act} --sparsity-ratio 0.0 \
+ --seed {42,137,256}
+
+ 6b. Sparsification — test robustness per activation
+ For act in {drelu, swiglu, geglu}, s in {0.3, 0.6}:
+ needle train --activation {act} --sparsity-ratio {s} \
+ --seed {42,137,256}
+
+Deliverables:
+ - Grouped bar chart per MRL dim: activation × compression → val PPL (Fig 6a)
+ - Relative degradation: (sparse_ppl / dense_ppl) per activation per dim
+ (Figure 6b) — KEY CHART: sparsification robustness not just raw quality
+ - Analysis: does dReLU's natural activation sparsity help or hurt at
+ smaller MRL dims where redundancy is already low?
+
+-------------------------------------------------------------------------------
+GROUP 7: MRL Dimension-Specific Analysis (Table 7, Figure 7)
+ *** MRL-SPECIFIC CONTRIBUTION ***
+-------------------------------------------------------------------------------
+Purpose: Deep-dive into what the model learns at each MRL dimension slice.
+
+Using best checkpoint from Group 2a:
+
+ 7a. Representation similarity across dims
+ - CKA (centered kernel alignment) between d'=1024 and each smaller d'
+ - Measure: do smaller dims learn a subspace of the full representation?
+
+ 7b. Per-dim generation quality
+ - Generate from each MRL dim slice at inference time
+ - Compare generation diversity, coherence, repetition rate
+
+ 7c. MRL loss weighting ablation
+ - Uniform weighting (current): L = (1/K) * sum(L_k)
+ - Scaled weighting: L = sum(d_k/D * L_k) (larger dims weighted more)
+ - Inverse weighting: L = sum(D/d_k * L_k) (smaller dims weighted more)
+
+Deliverables:
+ - CKA heatmap: dim × dim similarity (Figure 7a)
+ - Table: generation quality metrics at each dim
+ - Ablation table: MRL loss weighting strategy × dim → val PPL
+
+-------------------------------------------------------------------------------
+GROUP 8: Inference Efficiency Validation (Table 8, Figure 8)
+-------------------------------------------------------------------------------
+Purpose: Translate theoretical compression into real speedup across the
+full spectrum of deployment options.
+
+For each Pareto-optimal checkpoint from Groups 1-2, at each MRL dim:
+ needle test --checkpoint {path} --throughput-runs 50
+
+Measure at each deployment dim:
+ - Tokens/sec (decode throughput)
+ - Latency per generation (ms)
+ - Peak memory (via JAX profiler)
+
+Deliverables:
+ - Scatter plot: effective params vs actual throughput (Figure 8a)
+ - Table: all Pareto-optimal (s, d') with throughput + quality
+ - The "menu" figure: one chart showing ALL deployment options from a
+ single checkpoint, from largest (d'=1024, no pruning) to smallest
+ (d'=64, s=0.80) — THIS IS THE PAPER'S HERO FIGURE
+
+================================================================================
+SECTION 4 — EVALUATION PROTOCOL
+================================================================================
+
+For every checkpoint, evaluate at EACH MRL dimension slice:
+
+ 1. Validation perplexity (TinyStories val split)
+ - Full-precision val PPL
+ - INT4-quantized val PPL (at each MRL dim)
+ needle test --checkpoint {path} --max-eval-samples 5000
+
+ 2. Benchmark suite (at full dim and smallest viable dim)
+ needle evaluate --checkpoint {path} \
+ --benchmarks wikitext2 lambada hellaswag arc_easy --max-samples 500
+
+ 3. Generation quality at each MRL dim
+ needle test --checkpoint {path}
+ Report: bigram repetition rate, avg generation length, sample outputs
+
+ 4. Throughput at each MRL dim
+ needle test --checkpoint {path} --throughput-runs 50
+
+================================================================================
+SECTION 5 — EXPECTED TABLES AND FIGURES
+================================================================================
+
+Tables:
+ T1. Dense baselines + MRL scaling curve (val PPL at each dim)
+ T2. MRL + sparsification (sparsity × dim → val PPL)
+ T3. Recovery epoch scaling (epochs × MRL dims)
+ T4. QAT interaction (sparsity × {on,off} × MRL dims)
+ T5. Group size ablation (4 sizes × MRL dims)
+ T6. Activation × sparsification interaction (3 act × compression × MRL dims)
+ T7. MRL dimension analysis (CKA, generation quality, loss weighting)
+ T8. Inference efficiency (Pareto-optimal checkpoints × MRL dims)
+
+Figures:
+ F1. MRL scaling curve: deployment dim vs PPL + param count
+ F2. MRL + sparsity: (a) heatmap, (b) Pareto per dim — KEY FIGURE
+ F3. Recovery epoch scaling per MRL dim
+ F4. QAT interaction: sparsity × QAT × MRL dim
+ F5. Group size: (a) g vs PPL per dim, (b) effective bpp
+ F6. Activation × sparsification: (a) grouped bars per dim, (b) degradation
+ F7. MRL representation analysis: (a) CKA heatmap, (b) gen quality vs dim
+ F8. Hero figure: full deployment menu from single checkpoint
+
+================================================================================
+SECTION 6 — COMPUTE BUDGET
+================================================================================
+
+Per-run cost (MAIN-1024, 3 epochs, 1M samples, 4x TPU v5 lite):
+ ~15 minutes training + ~5 minutes MRL evaluation (5 dims × val + quant)
+
+Estimated total runs:
+ Group 1: 2 configs × 3 seeds = 6 runs
+ Group 2: 8 ratios × 3 seeds + 2 ratios × 3 seeds (rec)= 30 runs
+ Group 3: 5 epoch counts × 3 seeds = 15 runs
+ Group 4: 3 methods × 2 (QAT on/off) × 3 seeds = 18 runs
+ Group 5: 4 sizes × 2 sparsity × 3 seeds = 24 runs
+ Group 6: 3 act × (3 + 6) × 3 seeds = 27 runs
+ Group 7: 3 weighting strategies × 3 seeds = 9 runs
+ Group 8: evaluation only (reuses checkpoints) = 0 runs
+ ────────────────────────────────────────────────────────────────────
+ Total: 129 runs
+
+At ~20 min/run: ~43 TPU-hours (manageable on v5litepod-4)
+
+================================================================================
+SECTION 7 — ABLATION CHECKLIST
+================================================================================
+
+Before submission, verify these are addressed:
+
+ [ ] Statistical significance: 3 seeds, report mean +/- std
+ [ ] MRL scaling: smooth degradation curve from full dim to smallest
+ [ ] MRL + sparsity composability: both axes combine gracefully
+ [ ] Matched comparisons: compare methods at SAME effective param count
+ [ ] Recovery saturation: diminishing returns on fine-tuning epochs
+ [ ] QAT compatibility: sparsity + quantization + MRL are not antagonistic
+ [ ] Generation quality at each MRL dim, not just perplexity
+ [ ] Group size sensitivity: optimal g, interaction with MRL dim
+ [ ] Activation × sparsification × MRL dim: three-way interaction
+ [ ] MRL representation analysis: CKA, subspace structure
+ [ ] Honest limitations: sparse matmuls don't accelerate on current JAX;
+ MRL dimension reduction requires custom kernel for real speedup
+ [ ] Comparison to baselines: training K separate small models vs one MRL
+
+================================================================================
+SECTION 8 — EXECUTION ORDER
+================================================================================
+
+Phase 1 (Day 1): Groups 1 + 2 (36 runs, ~12 hrs)
+ Baselines and sparsification sweeps, all with MRL evaluation.
+ Deliverable: MRL scaling curves and sparsity Pareto frontiers.
+
+Phase 2 (Day 2): Groups 3 + 4 (33 runs, ~11 hrs)
+ Recovery and QAT ablations.
+ Deliverable: complete ablation tables.
+
+Phase 3 (Day 3): Groups 5 + 6 (51 runs, ~17 hrs)
+ Group size and activation interaction studies.
+ Deliverable: granularity and activation robustness analysis.
+
+Phase 4 (Day 4): Groups 7 + 8 + writing (9 runs + eval)
+ MRL analysis, inference benchmarks, paper draft.
+ Deliverable: hero figure, all tables populated.
+
+================================================================================
+SECTION 9 — PAPER NARRATIVE
+================================================================================
+
+Framing: "Train once, deploy anywhere."
+
+A single training run with Matryoshka loss produces a checkpoint that can be
+deployed at any dimension slice. Each slice can be further compressed with
+group-wise sparsification. The result is a deployment MENU:
+
+ ┌─────────────────────────────────────────────────────────────────────┐
+ │ Deployment option Est. params Val PPL Quant PPL │
+ │ ───────────────────────────────────────────────────────────────── │
+ │ d'=1024, full ~44M X.XX X.XX │
+ │ d'=512, full ~11M X.XX X.XX │
+ │ d'=512, s=0.3 ~7M X.XX X.XX │
+ │ d'=512, s=0.6 ~4M X.XX X.XX │
+ │ d'=256, full ~3M X.XX X.XX │
+ │ d'=256, s=0.3 ~2M X.XX X.XX │
+ │ d'=128, full ~0.8M X.XX X.XX │
+ │ d'=128, s=0.5 ~0.4M X.XX X.XX │
+ │ d'=64, full ~0.2M X.XX X.XX │
+ │ d'=64, s=0.8 ~0.04M X.XX X.XX │
+ └─────────────────────────────────────────────────────────────────────┘
+
+Key claims:
+ 1. MRL dimension pruning is nearly free — minimal degradation at 2x reduction
+ 2. Sparsification composes multiplicatively with MRL
+ 3. A single checkpoint spans 100x+ parameter range
+ 4. INT4 QAT is compatible with both compression axes
+ 5. MemoryMixer encoder is more compressible than standard transformer encoder
+
+================================================================================
diff --git a/experiements/datasets.txt b/experiements/datasets.txt
index e69de29..555dc0c 100644
--- a/experiements/datasets.txt
+++ b/experiements/datasets.txt
@@ -0,0 +1,26 @@
+================================================================================
+
+Task: Design & gather multimodal dataset, text tokenization, speech processing, and tool call strategy.
+Deliverable: A pipleine of preprocessed dataset, from which the codebase can stream datasets from.
+Assignee: Parkirat Sandhu & Jakub Mroz
+Deadline: 7th March
+
+================================================================================
+
+Text dataset: SYNTH (https://huggingface.co/datasets/PleIAs/SYNTH)
+Transcribe datasets: Emilia-Large, SPGI Speech
+Text tool call datasets: xlam-function-calling-60k, xlam-irrelevance-7.5k, Salesforce/APIGen-MT-5k, APIGen-Irrelevant-5k, glaive-function-calling, Team-ACE/ToolACE
+Voice tool call datasets: Use a good TTS model to generate voice commands from the tool call dataset prompts
+
+================================================================================
+
+Note:
+1. LEAP makes sense as we wish the dataset to preprocess and pretokenize the inputs before injesting into the model.
+2. We are training a cutom sentencepiece tokenizer with a small vocab size, given how constrained the decode is.
+3. Requires solid cloud infra experience, latency in injestion will slow down training.
+4. Record all obersvations, they will go into the paper!
+5. Design the batching (mix of speech and text and tools) experimantally.
+6. We should have a <transcribe> and <tools> tag for the two tasks we train on.
+
+================================================================================
+
diff --git a/experiements/pruning.txt b/experiements/pruning.txt
deleted file mode 100644
index 59828d7..0000000
--- a/experiements/pruning.txt
+++ /dev/null
@@ -1,492 +0,0 @@
-================================================================================
-MULTI-AXIS STRUCTURED COMPRESSION OF SMALL ENCODER-DECODER TRANSFORMERS:
-MATRYOSHKA DIMENSIONS, BLOCK SPARSITY, LAYER PRUNING, AND INT4 QAT
-================================================================================
-Target venue: ICLR 2026
-
-Thesis: Four complementary compression axes — Matryoshka dimension pruning,
-group-aligned block sparsity, layer removal, and INT4 quantization-aware
-training — can be jointly applied during training in a staged schedule to
-produce a single checkpoint that is deployable at multiple accuracy-efficiency
-operating points. Training once at d_model=D with MRL loss at dimensions
-{D, D/2, D/4, D/8, ...} yields a family of models from one training run,
-each further compressible via sparsification and layer pruning.
-
-================================================================================
-SECTION 1 — EXPERIMENTAL AXES
-================================================================================
-
-Independent variables:
- A. MRL deployment dim d' ∈ {1024, 512, 256, 128, 64}
- B. Sparsity ratio s ∈ {0.0, 0.10, 0.25, 0.33, 0.50}
- C. Layer prune ratio l ∈ {0.0, 0.125, 0.25, 0.375, 0.50}
- D. Group size g ∈ {16, 32, 64, 128}
- E. Pruning schedule ∈ {staged, simultaneous, reverse-order}
- F. FFN activation ∈ {dReLU, SwiGLU, GeGLU}
- G. Model scale ∈ {TOY-128, MAIN-1024}
- H. QAT ∈ {off, int4-group32}
-
-Dependent variables (measured at EACH MRL dimension):
- - Validation perplexity (TinyStories)
- - Quantized validation perplexity (INT4)
- - WikiText-2 perplexity
- - LAMBADA accuracy
- - HellaSwag accuracy
- - ARC-Easy accuracy
- - Estimated parameter count at deployment dim
- - Throughput (tokens/sec on inference)
- - Bigram repetition rate (generation diversity)
- - Block sparsity (% of fully-zero groups, group_size=g)
- - Effective FLOPs (est. params × density)
-
-================================================================================
-SECTION 2 — MODEL CONFIGURATIONS
-================================================================================
-
-Architecture: MemoryMixer encoder + Transformer decoder, shared embedding,
-GQA (grouped query attention), ZC-RMSNorm, RoPE, SentencePiece BPE (8192).
-
-TOY-128 (rapid iteration):
- mrl_dims=[128, 64], d_model=128, num_heads=4, num_kv_heads=2,
- num_enc_layers=2, num_dec_layers=2, d_ff=512,
- max_enc_len=128, max_dec_len=128, vocab=8192, dtype=bfloat16
- needle train --toy --seed {42,137,256}
-
-MAIN-1024 (paper results):
- mrl_dims=[1024, 512, 256, 128, 64], d_model=1024, num_heads=4,
- num_kv_heads=2, num_enc_layers=4, num_dec_layers=2, d_ff=4096,
- max_enc_len=256, max_dec_len=256, vocab=8192, dtype=bfloat16
- needle train --seed {42,137,256}
-
-Both trained on TinyStories (1M samples, encoder-decoder split at 30%).
-Optimizer: Muon (kernels) + AdamW (embeddings/norms), WSD schedule.
-All experiments report metrics at each MRL dimension slice.
-
-================================================================================
-SECTION 3 — EXPERIMENT GROUPS
-================================================================================
-
-All experiments use 3 seeds (42, 137, 256) for statistical significance.
-Report mean +/- std on all metrics. Every run automatically reports val PPL
-and INT4-quantized val PPL at each MRL dimension slice.
-
--------------------------------------------------------------------------------
-GROUP 1: Dense Baselines + MRL Dimension Scaling (Table 1, Figure 1)
--------------------------------------------------------------------------------
-Purpose: Establish quality at each deployment dimension from a single training
-run. This is the MRL baseline — no sparsification or layer pruning.
-
- 1a. MAIN-1024, 3 epochs, no pruning, MRL dims [1024,512,256,128,64]
- needle train --sparsity-ratio 0.0 --layer-prune-ratio 0.0 \
- --seed {42,137,256}
-
- 1b. TOY-128, 3 epochs, no pruning, MRL dims [128,64]
- needle train --toy --sparsity-ratio 0.0 --layer-prune-ratio 0.0 \
- --seed {42,137,256}
-
-Deliverables:
- - Table: val PPL and quant PPL at each MRL dim (1024, 512, 256, 128, 64)
- - Line plot: deployment dim vs val PPL — the MRL scaling curve (Figure 1a)
- - Line plot: deployment dim vs estimated param count (Figure 1b)
- - Benchmark scores at each dim
- - These are the denominators for all relative comparisons
-
--------------------------------------------------------------------------------
-GROUP 2: MRL + Group-Wise Sparsification (Table 2, Figure 2)
--------------------------------------------------------------------------------
-Purpose: Characterise how MRL dimension scaling interacts with block sparsity.
-At smaller deployment dims, does sparsification hurt more or less?
-
-Schedule: Epoch 1 dense, epochs 2-3 with sparsity mask locked.
-
- For s in {0.10, 0.25, 0.33, 0.50}:
- 2a. MAIN-1024, 3 epochs, --sparsity-ratio s
- needle train --sparsity-ratio {s} --layer-prune-ratio 0.0 \
- --seed {42,137,256}
-
-Deliverables:
- - 2D heatmap: (sparsity, MRL dim) → val PPL (Figure 2a)
- - Pareto curve per MRL dim: sparsity % vs val PPL (Figure 2b)
- - Table: all metrics at each (sparsity, dim) pair
- - Analysis: does sparsification degrade smaller MRL slices more?
- (Hypothesis: smaller slices are more sensitive because fewer dimensions
- carry the representation — each zeroed weight matters more)
-
--------------------------------------------------------------------------------
-GROUP 3: MRL + Layer Pruning (Table 3, Figure 3)
--------------------------------------------------------------------------------
-Purpose: Characterise how MRL dimension scaling interacts with layer removal.
-
-Schedule: Epoch 1 dense, epochs 2-3 with reduced architecture.
-
- For l in {0.125, 0.25, 0.375, 0.50}:
- 3a. MAIN-1024 (4 enc + 2 dec layers), 3 epochs, --layer-prune-ratio l
- needle train --sparsity-ratio 0.0 --layer-prune-ratio {l} \
- --seed {42,137,256}
- Expected surviving layers: 5, 4-5, 4, 3 (of 6 total)
-
-Deliverables:
- - 2D heatmap: (layer ratio, MRL dim) → val PPL (Figure 3a)
- - Pareto curve: param count vs val PPL at each MRL dim (Figure 3b)
- - Per-run encoder/decoder surviving layer counts
- - Analysis: does the pruner prefer removing encoder or decoder layers?
- (Hypothesis: encoder MemoryMixer blocks are pruned more aggressively
- since the decoder carries more of the generation burden)
-
--------------------------------------------------------------------------------
-GROUP 4: Full Pipeline — MRL + Sparsification + Layer Pruning (Table 4, Fig 4)
- *** CORE CONTRIBUTION ***
--------------------------------------------------------------------------------
-Purpose: Show that all three compression axes compose — a single checkpoint
-trained with MRL loss, then sparsified and layer-pruned, produces a family
-of models spanning 10-100x parameter range. This is the central claim.
-
-Schedule (staged):
- Epoch 1: dense training with MRL loss
- Epoch 2: apply group-wise sparsification, train with mask + MRL
- Epoch 3: apply layer pruning, train reduced architecture + mask + MRL
- (Optional epoch 4: recovery fine-tuning)
-
- For (s, l) in grid:
- {(0.10, 0.125), (0.10, 0.25),
- (0.25, 0.125), (0.25, 0.25), (0.25, 0.375),
- (0.33, 0.125), (0.33, 0.25),
- (0.50, 0.25), (0.50, 0.50)}
-
- 4a. MAIN-1024, 3 epochs
- needle train --sparsity-ratio {s} --layer-prune-ratio {l} \
- --seed {42,137,256}
-
- 4b. Select best 4 (s,l) pairs from 4a, run with 4 epochs (recovery)
- needle train --epochs 4 --sparsity-ratio {s} --layer-prune-ratio {l} \
- --seed {42,137,256}
-
-Deliverables:
- - 3D surface: (s, l, MRL dim) → val PPL (Figure 4a, sliced by dim)
- - Pareto overlay at each MRL dim: full-pipeline vs sparsity-only vs
- layer-only at matched effective params (Figure 4b) — KEY FIGURE
- - Table: full metrics for all (s, l) pairs at each MRL dim
- - Compression spectrum: for best (s, l), show the range of models:
- d'=1024 (full) → d'=64 (smallest) with estimated params and PPL
- - Analysis: multiplicative compression — MRL dimension reduction ×
- sparsity × layer removal = total compression ratio
-
--------------------------------------------------------------------------------
-GROUP 5: Schedule Ablation (Table 5)
--------------------------------------------------------------------------------
-Purpose: Justify the staged schedule (sparsify first, then prune layers).
-
-Pick the best (s, l) pair from Group 4a. Run 3 schedules:
-
- 5a. Staged (default): sparsify epoch 2, layer prune epoch 3
- needle train --sparsity-ratio {s} --layer-prune-ratio {l}
-
- 5b. Reverse: layer prune epoch 2, sparsify epoch 3
- [Requires code change to swap epoch triggers]
-
- 5c. Simultaneous: both at epoch 2, recover epoch 3
- [Requires code change to apply both at same epoch]
-
-Deliverables:
- - Table: val PPL at each MRL dim for each schedule
- - Analysis: does schedule affect small MRL dims differently from large?
-
--------------------------------------------------------------------------------
-GROUP 6: Recovery Epoch Scaling (Figure 5)
--------------------------------------------------------------------------------
-Purpose: How many recovery epochs are needed after compression?
-
-Pick the best (s, l) pair from Group 4a. Vary total epochs:
-
- For E in {2, 3, 4, 5, 6}:
- 6a. MAIN-1024, E epochs, --sparsity-ratio {s} --layer-prune-ratio {l}
- (pruning at epochs 2 and 3; recovery = E - 3 epochs)
-
-Deliverables:
- - Line plot per MRL dim: recovery epochs vs val PPL (Figure 5)
- - Analysis: do smaller MRL dims need more recovery?
-
--------------------------------------------------------------------------------
-GROUP 7: QAT Interaction Study (Table 6)
--------------------------------------------------------------------------------
-Purpose: Show that group-wise sparsification is compatible with INT4 QAT
-because both operate on the same group_size=32 granularity. Measure at
-each MRL dimension — INT4 quantization error at smaller dims may compound.
-
- 7a. Dense + QAT (baseline, = Group 1a)
- 7b. Sparsified (s=0.25) + QAT
- 7c. Layer-pruned (l=0.25) + QAT
- 7d. Joint (s=0.25, l=0.25) + QAT
-
- For comparison, also run 7b-7d WITHOUT QAT by modifying _train_step
- to not call _quantize_params.
-
-Deliverables:
- - Table: (QAT on/off) × compression method × MRL dim → val PPL
- - Analysis: does QAT degrade more at smaller MRL dims?
- (Hypothesis: yes — fewer dimensions means each quantization error
- has proportionally larger impact on output quality)
- - Effective compression: MRL dim reduction × sparsity × INT4 → total
- bits per effective parameter
-
--------------------------------------------------------------------------------
-GROUP 8: Group Size Ablation (Table 7, Figure 6)
--------------------------------------------------------------------------------
-Purpose: Determine optimal block granularity for sparsification.
-
-Fixed: MAIN-1024, s=0.25, 3 epochs, QAT on.
-
- For g in {16, 32, 64, 128}:
- 8a. Sparsification only
- needle train --sparsity-ratio 0.25 --layer-prune-ratio 0.0 \
- --group-size {g} --seed {42,137,256}
-
- 8b. Joint sparsification + layer pruning (best l from Group 4a)
- needle train --sparsity-ratio 0.25 --layer-prune-ratio {l} \
- --group-size {g} --seed {42,137,256}
-
- High sparsity stress test:
- For g in {16, 32, 64, 128}:
- 8c. s=0.50, sparsification only
- needle train --sparsity-ratio 0.50 --layer-prune-ratio 0.0 \
- --group-size {g} --seed {42,137,256}
-
-Deliverables:
- - Line plot per MRL dim: group size vs val PPL (Figure 6a)
- - Table: all metrics at each (group size, MRL dim)
- - Effective bits-per-param: bpp = (1 - sparsity) * (4 + 32/g)
- - Analysis: does optimal group size change with MRL deployment dim?
-
--------------------------------------------------------------------------------
-GROUP 9: Activation Function x Pruning Interaction (Table 8, Figure 7)
- *** KEY FINDING ***
--------------------------------------------------------------------------------
-Purpose: Test whether activation function choice determines pruning robustness
-across MRL dimension slices.
-
-Activation definitions:
- - dReLU: ReLU(gate) * ReLU(up) — naturally ~75% sparse activations
- - SwiGLU: SiLU(gate) * up — smooth, dense activations
- - GeGLU: GELU(gate) * up — smooth, nearly dense activations
-
- 9a. Dense baselines (no pruning) — isolate activation quality
- For act in {drelu, swiglu, geglu}:
- needle train --activation {act} --sparsity-ratio 0.0 \
- --layer-prune-ratio 0.0 --seed {42,137,256}
-
- 9b. Sparsification only — test pruning robustness per activation
- For act in {drelu, swiglu, geglu}, s in {0.25, 0.50}:
- needle train --activation {act} --sparsity-ratio {s} \
- --layer-prune-ratio 0.0 --seed {42,137,256}
-
- 9c. Joint — full compression
- For act in {drelu, swiglu, geglu}:
- needle train --activation {act} --sparsity-ratio 0.25 \
- --layer-prune-ratio 0.25 --seed {42,137,256}
-
-Deliverables:
- - Grouped bar chart per MRL dim: activation × compression → val PPL (Fig 7a)
- - Relative degradation: (pruned_ppl / dense_ppl) per activation per dim
- (Figure 7b) — KEY CHART: pruning robustness not just raw quality
- - Analysis: does dReLU's natural activation sparsity help or hurt at
- smaller MRL dims where redundancy is already low?
-
--------------------------------------------------------------------------------
-GROUP 10: MRL Dimension-Specific Analysis (Table 9, Figure 8)
- *** MRL-SPECIFIC CONTRIBUTION ***
--------------------------------------------------------------------------------
-Purpose: Deep-dive into what the model learns at each MRL dimension slice.
-
-Using best checkpoint from Group 4a:
-
- 10a. Representation similarity across dims
- - CKA (centered kernel alignment) between d'=1024 and each smaller d'
- - Measure: do smaller dims learn a subspace of the full representation?
-
- 10b. Per-dim generation quality
- - Generate from each MRL dim slice at inference time
- - Compare generation diversity, coherence, repetition rate
-
- 10c. MRL loss weighting ablation
- - Uniform weighting (current): L = (1/K) * sum(L_k)
- - Scaled weighting: L = sum(d_k/D * L_k) (larger dims weighted more)
- - Inverse weighting: L = sum(D/d_k * L_k) (smaller dims weighted more)
-
-Deliverables:
- - CKA heatmap: dim × dim similarity (Figure 8a)
- - Table: generation quality metrics at each dim
- - Ablation table: MRL loss weighting strategy × dim → val PPL
-
--------------------------------------------------------------------------------
-GROUP 11: Inference Efficiency Validation (Table 10, Figure 9)
--------------------------------------------------------------------------------
-Purpose: Translate theoretical compression into real speedup across the
-full spectrum of deployment options.
-
-For each Pareto-optimal checkpoint from Groups 2-4, at each MRL dim:
- needle test --checkpoint {path} --throughput-runs 50
-
-Measure at each deployment dim:
- - Tokens/sec (decode throughput)
- - Latency per generation (ms)
- - Peak memory (via JAX profiler)
-
-Deliverables:
- - Scatter plot: effective params vs actual throughput (Figure 9a)
- - Table: all Pareto-optimal (s, l, d') with throughput + quality
- - The "menu" figure: one chart showing ALL deployment options from a
- single checkpoint, from largest (d'=1024, no pruning) to smallest
- (d'=64, s=0.50, l=0.50) — THIS IS THE PAPER'S HERO FIGURE
-
-================================================================================
-SECTION 4 — EVALUATION PROTOCOL
-================================================================================
-
-For every checkpoint, evaluate at EACH MRL dimension slice:
-
- 1. Validation perplexity (TinyStories val split)
- - Full-precision val PPL
- - INT4-quantized val PPL (at each MRL dim)
- needle test --checkpoint {path} --max-eval-samples 5000
-
- 2. Benchmark suite (at full dim and smallest viable dim)
- needle evaluate --checkpoint {path} \
- --benchmarks wikitext2 lambada hellaswag arc_easy --max-samples 500
-
- 3. Generation quality at each MRL dim
- needle test --checkpoint {path}
- Report: bigram repetition rate, avg generation length, sample outputs
-
- 4. Throughput at each MRL dim
- needle test --checkpoint {path} --throughput-runs 50
-
-================================================================================
-SECTION 5 — EXPECTED TABLES AND FIGURES
-================================================================================
-
-Tables:
- T1. Dense baselines + MRL scaling curve (val PPL at each dim)
- T2. MRL + sparsification (sparsity × dim → val PPL)
- T3. MRL + layer pruning (layer ratio × dim → val PPL)
- T4. Full pipeline grid (s × l at each MRL dim)
- T5. Schedule ablation (3 schedules × MRL dims)
- T6. QAT interaction (4 methods × {on,off} × MRL dims)
- T7. Group size ablation (4 sizes × MRL dims)
- T8. Activation × pruning interaction (3 act × compression × MRL dims)
- T9. MRL dimension analysis (CKA, generation quality, loss weighting)
- T10. Inference efficiency (Pareto-optimal checkpoints × MRL dims)
-
-Figures:
- F1. MRL scaling curve: deployment dim vs PPL + param count
- F2. MRL + sparsity: (a) heatmap, (b) Pareto per dim
- F3. MRL + layer pruning: (a) heatmap, (b) Pareto per dim
- F4. Full pipeline: (a) 3D surface slices, (b) Pareto overlay — KEY FIGURE
- F5. Recovery epoch scaling per MRL dim
- F6. Group size: (a) g vs PPL per dim, (b) effective bpp
- F7. Activation × pruning: (a) grouped bars per dim, (b) degradation ratio
- F8. MRL representation analysis: (a) CKA heatmap, (b) gen quality vs dim
- F9. Hero figure: full deployment menu from single checkpoint
-
-================================================================================
-SECTION 6 — COMPUTE BUDGET
-================================================================================
-
-Per-run cost (MAIN-1024, 3 epochs, 1M samples, 4x TPU v5 lite):
- ~15 minutes training + ~5 minutes MRL evaluation (5 dims × val + quant)
-
-Estimated total runs:
- Group 1: 2 configs × 3 seeds = 6 runs
- Group 2: 4 ratios × 3 seeds = 12 runs
- Group 3: 4 ratios × 3 seeds = 12 runs
- Group 4: 9 pairs × 3 seeds + 4 pairs × 3 seeds (recov)= 39 runs
- Group 5: 3 schedules × 3 seeds = 9 runs
- Group 6: 5 epoch counts × 3 seeds = 15 runs
- Group 7: 4 methods × 2 (QAT on/off) × 3 seeds = 24 runs
- Group 8: 4 sizes × 3 configs × 3 seeds = 36 runs
- Group 9: 3 act × (3 + 6 + 3) × 3 seeds = 36 runs
- Group 10: 3 weighting strategies × 3 seeds = 9 runs
- Group 11: evaluation only (reuses checkpoints) = 0 runs
- ────────────────────────────────────────────────────────────────────
- Total: 198 runs
-
-At ~20 min/run: ~66 TPU-hours (manageable on v5litepod-4)
-
-================================================================================
-SECTION 7 — ABLATION CHECKLIST
-================================================================================
-
-Before submission, verify these are addressed:
-
- [ ] Statistical significance: 3 seeds, report mean +/- std
- [ ] MRL scaling: smooth degradation curve from full dim to smallest
- [ ] MRL + compression composability: all three axes combine gracefully
- [ ] Matched comparisons: compare methods at SAME effective param count
- [ ] Schedule sensitivity: staged vs reverse vs simultaneous
- [ ] Recovery saturation: diminishing returns on fine-tuning epochs
- [ ] QAT compatibility: sparsity + quantization + MRL are not antagonistic
- [ ] Pruning bias: report which layers (enc vs dec) get pruned
- [ ] Generation quality at each MRL dim, not just perplexity
- [ ] Group size sensitivity: optimal g, interaction with MRL dim
- [ ] Activation × pruning × MRL dim: three-way interaction
- [ ] MRL representation analysis: CKA, subspace structure
- [ ] Honest limitations: sparse matmuls don't accelerate on current JAX;
- MRL dimension reduction requires custom kernel for real speedup
- [ ] Comparison to baselines: training K separate small models vs one MRL
-
-================================================================================
-SECTION 8 — EXECUTION ORDER
-================================================================================
-
-Phase 1 (Day 1): Groups 1 + 2 + 3 (30 runs, ~10 hrs)
- Baselines, sparsity-only, layer-prune-only, all with MRL evaluation.
- Deliverable: MRL scaling curves and single-axis Pareto frontiers.
-
-Phase 2 (Day 2): Group 4 (39 runs, ~13 hrs)
- Full pipeline grid. This produces the key result.
- Deliverable: multi-axis Pareto and deployment menu.
-
-Phase 3 (Day 3): Groups 5 + 6 + 7 (48 runs, ~16 hrs)
- Schedule, recovery, QAT ablations.
- Deliverable: complete ablation tables.
-
-Phase 4 (Day 4): Groups 8 + 9 (72 runs, ~24 hrs)
- Group size and activation interaction studies.
- Deliverable: granularity and activation robustness analysis.
-
-Phase 5 (Day 5): Groups 10 + 11 + writing (9 runs + eval)
- MRL analysis, inference benchmarks, paper draft.
- Deliverable: hero figure, all tables populated.
-
-================================================================================
-SECTION 9 — PAPER NARRATIVE
-================================================================================
-
-Framing: "Train once, deploy anywhere."
-
-A single training run with Matryoshka loss produces a checkpoint that can be
-deployed at any dimension slice. Each slice can be further compressed with
-group-wise sparsification and layer pruning. The result is a deployment MENU:
-
- ┌─────────────────────────────────────────────────────────────────────┐
- │ Deployment option Est. params Val PPL Quant PPL │
- │ ───────────────────────────────────────────────────────────────── │
- │ d'=1024, full ~44M X.XX X.XX │
- │ d'=512, full ~11M X.XX X.XX │
- │ d'=512, s=0.33 ~7M X.XX X.XX │
- │ d'=256, full ~3M X.XX X.XX │
- │ d'=256, s=0.33 ~2M X.XX X.XX │
- │ d'=256, s=0.33, l=0.25 ~1.5M X.XX X.XX │
- │ d'=128, full ~0.8M X.XX X.XX │
- │ d'=128, s=0.33 ~0.5M X.XX X.XX │
- │ d'=64, full ~0.2M X.XX X.XX │
- │ d'=64, s=0.33, l=0.25 ~0.1M X.XX X.XX │
- └─────────────────────────────────────────────────────────────────────┘
-
-Key claims:
- 1. MRL dimension pruning is nearly free — minimal degradation at 2x reduction
- 2. Sparsification and layer pruning compose multiplicatively with MRL
- 3. A single checkpoint spans 100x+ parameter range
- 4. INT4 QAT is compatible with all three compression axes
- 5. MemoryMixer encoder is more compressible than standard transformer encoder
-
-================================================================================
diff --git a/experiements/tokenizer.txt b/experiements/tokenizer.txt
deleted file mode 100644
index e69de29..0000000
diff --git a/experiements/training.txt b/experiements/training.txt
new file mode 100644
index 0000000..706b729
--- /dev/null
+++ b/experiements/training.txt
@@ -0,0 +1,6 @@
+================================================================================
+Task: Design the training strategy & loss functions for training a transcribe+tools model.
+Deliverable: A well-executed ablation of all strategies against SOTA.
+Assignee: Karen Mosoyan & Satyajit Kumar
+Deadline: 7th March
+================================================================================
diff --git a/src/cli.py b/src/cli.py
index 51803d6..b8b2255 100644
--- a/src/cli.py
+++ b/src/cli.py
@@ -11,7 +11,7 @@ HELP = """
│ │
│ train │
│ --toy Use toy config for quick iteration │
- │ --epochs INT Training epochs (default: 2) │
+ │ --epochs INT Training epochs (default: 1) │
│ --batch-size INT Batch size (default: 32) │
│ --lr FLOAT AdamW learning rate (default: 3e-4) │
│ --muon-lr FLOAT Muon learning rate (default: 0.02) │
@@ -26,6 +26,9 @@ HELP = """
│ --sparsity-ratio FLOAT Block prune ratio (default: 0.33) │
│ --layer-prune-ratio FL Layer prune ratio (default: 0.0) │
│ --group-size INT Quant/prune group size (default: 32) │
+ │ --prune-interval INT Steps between mask updates (def: 100) │
+ │ --prune-start-frac FL Start pruning at this frac (def: 0.33) │
+ │ --prune-end-frac FL Lock mask at this frac (def: 0.67) │
│ --activation STR drelu|swiglu|geglu (default: drelu) │
│ --warmup-ratio FLOAT LR warmup ratio (default: 0.05) │
│ --eval-every INT Val eval interval (default: 1000) │
@@ -34,11 +37,6 @@ HELP = """
│ --checkpoint-dir DIR Checkpoint directory │
│ --seed INT Random seed (default: 42) │
│ │
- │ sweep │
- │ --sweep-config PATH Sweep YAML config │
- │ --project STR W&B project name (default: needle-v1) │
- │ --count INT Number of trials (default: 20) │
- │ │
│ run │
│ --checkpoint PATH Path to model checkpoint (required) │
│ --prompts STR [...] One or more prompts to continue │
@@ -85,11 +83,11 @@ TOY_CONFIG = {
MAIN_CONFIG = {
"num_heads": 4,
- "num_layers": 12,
+ "num_layers": 24,
"num_dec_layers": 4,
"max_enc_len": 256,
"max_dec_len": 256,
- "max_samples": 1000000,
+ "max_samples": None,
}
@@ -113,7 +111,7 @@ def main():
p = sub.add_parser("train", add_help=False)
p.add_argument("--toy", action="store_true")
p.add_argument("--checkpoint", type=str, default=None)
- p.add_argument("--epochs", type=int, default=2)
+ p.add_argument("--epochs", type=int, default=1)
p.add_argument("--batch-size", type=int, default=32)
p.add_argument("--lr", type=float, default=3e-4)
p.add_argument("--muon-lr", type=float, default=0.02)
@@ -131,13 +129,19 @@ def main():
p.add_argument("--seed", type=int, default=42)
p.add_argument("--eval-every", type=int, default=1000)
p.add_argument("--max-eval-samples", type=int, default=None)
- p.add_argument("--sparsity-ratio", type=float, default=0.33)
+ p.add_argument("--sparsity-ratio", type=float, default=0.5)
p.add_argument("--layer-prune-ratio", type=float, default=0.0)
p.add_argument("--group-size", type=int, default=32)
+ p.add_argument("--prune-interval", type=int, default=100,
+ help="Steps between mask updates during gradual pruning (default: 100)")
+ p.add_argument("--prune-start-frac", type=float, default=0.33,
+ help="Fraction of epoch to train before starting gradual pruning (default: 0.33)")
+ p.add_argument("--prune-end-frac", type=float, default=0.67,
+ help="Fraction of epoch at which pruning finishes and mask locks (default: 0.67)")
p.add_argument("--activation", type=str, default="drelu", choices=["drelu", "swiglu", "geglu"])
p.add_argument("--num-memory-slots", type=int, default=64)
- p.add_argument("--mrl-dims", type=int, nargs="*", default=[512, 256, 128, 64],
- help="MRL dimension pruning targets (default: 512 256 128 64)")
+ p.add_argument("--mrl-dims", type=int, nargs="*", default=[1024, 512, 256, 128, 64],
+ help="MRL dimension pruning targets (default: 1024 512 256 128 64)")
p = sub.add_parser("run", add_help=False)
p.add_argument("--checkpoint", type=str, required=True)
@@ -156,25 +160,6 @@ def main():
p.add_argument("--temperature", type=float, default=0.8)
p.add_argument("--throughput-runs", type=int, default=10)
- p = sub.add_parser("sweep", add_help=False)
- p.add_argument("--sweep-config", type=str, default="sweep.yaml")
- p.add_argument("--project", type=str, default="needle-v1")
- p.add_argument("--count", type=int, default=20)
- p.add_argument("--epochs", type=int, default=1)
- p.add_argument("--batch-size", type=int, default=16)
- p.add_argument("--lr", type=float, default=3e-4)
- p.add_argument("--muon-lr", type=float, default=0.02)
- p.add_argument("--d-model", type=int, default=128)
- p.add_argument("--num-heads", type=int, default=4)
- p.add_argument("--num-layers", type=int, default=2)
- p.add_argument("--max-enc-len", type=int, default=128)
- p.add_argument("--max-dec-len", type=int, default=128)
- p.add_argument("--max-samples", type=int, default=20000)
- p.add_argument("--warmup-ratio", type=float, default=0.05)
- p.add_argument("--dtype", type=str, default="bfloat16", choices=["float32", "bfloat16"])
- p.add_argument("--checkpoint-dir", type=str, default="checkpoints")
- p.add_argument("--seed", type=int, default=42)
-
p = sub.add_parser("evaluate", add_help=False)
p.add_argument("--checkpoint", type=str, required=True)
p.add_argument("--benchmarks", type=str, nargs="*",
@@ -222,9 +207,6 @@ def main():
_apply_train_defaults(args)
from .train import train
train(args)
- elif args.command == "sweep":
- from .train import sweep
- sweep(args)
elif args.command == "run":
from .run import main as run_main
run_main(args)
diff --git a/src/data.py b/src/data.py
index 7c572b8..9cadde8 100644
--- a/src/data.py
+++ b/src/data.py
@@ -1,4 +1,5 @@
import hashlib
+import multiprocessing as mp
import os
os.environ["TRANSFORMERS_NO_ADVISORY_WARNINGS"] = "1"
os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1"
@@ -63,6 +64,23 @@ class NeedleTokenizer:
return {"input_ids": all_ids}
+_worker_sp = None
+_worker_max_len = None
+
+
+def _init_worker(model_path, max_length):
+ """Initializer for multiprocessing pool — loads SP model once per worker."""
+ global _worker_sp, _worker_max_len
+ _worker_sp = spm.SentencePieceProcessor()
+ _worker_sp.Load(model_path)
+ _worker_max_len = max_length
+
+
+def _tokenize_chunk(texts):
+ """Encode a chunk of texts in a worker process."""
+ return [_worker_sp.Encode(t, out_type=int)[:_worker_max_len] for t in texts]
+
+
def train_tokenizer(vocab_size=8192, max_samples=None):
"""Train a SentencePiece BPE tokenizer on TinyStories."""
model_path = TOKENIZER_PREFIX + ".model"
@@ -151,47 +169,54 @@ def prepare_encoder_decoder_pairs(ds, tokenizer, max_enc_len=128, max_dec_len=12
bos_id = tokenizer.eos_token_id
max_total = max_enc_len + max_dec_len
- print("Tokenizing...")
+ # --- Parallel tokenization ---
texts = list(ds["text"])
- all_tokens = tokenizer(
- texts, truncation=True, max_length=max_total,
- )["input_ids"]
-
- n = len(all_tokens)
- enc_inputs = np.full((n, max_enc_len), pad_id, dtype=np.int32)
- dec_inputs = np.full((n, max_dec_len), pad_id, dtype=np.int32)
- dec_targets = np.full((n, max_dec_len), pad_id, dtype=np.int32)
-
- kept = 0
- for tokens in tqdm(all_tokens, desc="Building pairs"):
- if len(tokens) < 6:
- continue
-
- split_point = max(2, int(len(tokens) * split_ratio))
- enc_tokens = tokens[:split_point][:max_enc_len]
- dec_tokens = tokens[split_point:][:max_dec_len]
-
- if len(dec_tokens) < 2:
- continue
-
- enc_len = len(enc_tokens)
- dec_len = len(dec_tokens)
- enc_inputs[kept, :enc_len] = enc_tokens
- dec_inputs[kept, 0] = bos_id
- dec_inputs[kept, 1:dec_len] = dec_tokens[:dec_len - 1]
- dec_targets[kept, :dec_len] = dec_tokens
- kept += 1
-
- enc_inputs = enc_inputs[:kept]
- dec_inputs = dec_inputs[:kept]
- dec_targets = dec_targets[:kept]
+ num_workers = min(os.cpu_count() or 1, 8)
+ model_path = TOKENIZER_PREFIX + ".model"
+ chunk_size = max(1, len(texts) // (num_workers * 4))
+ chunks = [texts[i:i + chunk_size] for i in range(0, len(texts), chunk_size)]
+
+ print(f"Tokenizing ({num_workers} workers)...")
+ with mp.Pool(num_workers, initializer=_init_worker,
+ initargs=(model_path, max_total)) as pool:
+ results = pool.map(_tokenize_chunk, chunks)
+ all_tokens = [tok for chunk in results for tok in chunk]
+
+ # --- Pre-filter by length to reduce loop iterations ---
+ lengths = np.array([len(t) for t in all_tokens])
+ valid_mask = lengths >= 6
+ valid_indices = np.where(valid_mask)[0]
+
+ split_points = np.maximum(2, (lengths[valid_indices] * split_ratio).astype(np.int32))
+ enc_lens = np.minimum(split_points, max_enc_len)
+ dec_lens = np.minimum(lengths[valid_indices] - split_points, max_dec_len)
+ keep = dec_lens >= 2
+
+ valid_indices = valid_indices[keep]
+ split_points = split_points[keep]
+ enc_lens = enc_lens[keep]
+ dec_lens = dec_lens[keep]
+
+ n_keep = len(valid_indices)
+ enc_inputs = np.full((n_keep, max_enc_len), pad_id, dtype=np.int32)
+ dec_inputs = np.full((n_keep, max_dec_len), pad_id, dtype=np.int32)
+ dec_targets = np.full((n_keep, max_dec_len), pad_id, dtype=np.int32)
+
+ for i, (idx, sp, el, dl) in enumerate(
+ zip(valid_indices, split_points, enc_lens, dec_lens)
+ ):
+ tokens = all_tokens[idx]
+ enc_inputs[i, :el] = tokens[:sp][:max_enc_len]
+ dec_inputs[i, 0] = bos_id
+ dec_inputs[i, 1:dl] = tokens[sp:][:dl - 1]
+ dec_targets[i, :dl] = tokens[sp:][:max_dec_len]
# Save cache
os.makedirs(CACHE_DIR, exist_ok=True)
np.save(cache_path + "_enc.npy", enc_inputs)
np.save(cache_path + "_dec_in.npy", dec_inputs)
np.save(cache_path + "_dec_tgt.npy", dec_targets)
- print(f"Cached data to {CACHE_DIR}/{cache_id}")
+ print(f"Cached {n_keep:,} pairs to {CACHE_DIR}/{cache_id}")
return enc_inputs, dec_inputs, dec_targets
diff --git a/src/train.py b/src/train.py
index d46b551..7d4112c 100644
--- a/src/train.py
+++ b/src/train.py
@@ -205,6 +205,56 @@ def _block_prune_params(params, prune_ratio=0.25, group_size=32):
return jax.tree_util.tree_map_with_path(_prune_weight, params)
+def _cubic_sparsity_schedule(step, t_start, t_end, s_final):
+ """Cubic sparsity ramp (Zhu & Gupta 2017). Returns target sparsity at *step*."""
+ if step < t_start:
+ return 0.0
+ if step >= t_end:
+ return s_final
+ frac = (step - t_start) / (t_end - t_start)
+ return s_final * (1.0 - (1.0 - frac) ** 3)
+
+
+def _make_prune_mask(params, sparsity, group_size):
+ """Compute block-prune mask entirely on-device (no numpy round-trips).
+
+ Returns (pruned_params, binary_mask).
+ """
+ # Pass 1: collect block L1 scores as JAX arrays
+ all_scores = []
+ for _, leaf in jax.tree_util.tree_leaves_with_path(params):
+ if leaf.ndim != 2:
+ continue
+ in_feat, out_feat = leaf.shape
+ gs = min(group_size, in_feat)
+ pad = (gs - in_feat % gs) % gs
+ w = jnp.pad(leaf, ((0, pad), (0, 0))) if pad else leaf
+ scores = jnp.sum(jnp.abs(w.reshape(-1, gs, out_feat)), axis=1).ravel()
+ all_scores.append(scores)
+
+ if not all_scores:
+ ones = jax.tree.map(jnp.ones_like, params)
+ return params, ones
+
+ threshold = jnp.percentile(jnp.concatenate(all_scores), sparsity * 100)
+
+ # Pass 2: build per-leaf binary mask
+ def _leaf_mask(path, leaf):
+ if leaf.ndim != 2:
+ return jnp.ones_like(leaf)
+ in_feat, out_feat = leaf.shape
+ gs = min(group_size, in_feat)
+ pad = (gs - in_feat % gs) % gs
+ w = jnp.pad(leaf, ((0, pad), (0, 0))) if pad else leaf
+ w_grouped = w.reshape(-1, gs, out_feat)
+ block_keep = (jnp.sum(jnp.abs(w_grouped), axis=1, keepdims=True) > threshold)
+ return jnp.broadcast_to(block_keep, w_grouped.shape).reshape(-1, out_feat)[:in_feat].astype(jnp.float32)
+
+ mask = jax.tree_util.tree_map_with_path(_leaf_mask, params)
+ pruned = jax.tree.map(lambda w, m: w * m, params, mask)
+ return pruned, mask
+
+
def _layer_prune(params, config, layer_prune_ratio=0.25):
"""Remove the lowest-scoring encoder/decoder blocks by L1 magnitude.
@@ -529,37 +579,22 @@ def train(args):
sparsity_ratio = getattr(args, "sparsity_ratio", 0.0)
layer_prune_ratio = getattr(args, "layer_prune_ratio", 0.0)
prune_mask = None
+ gradual_pruning_done = False
- weight_prune_epoch = 1 if sparsity_ratio > 0 else -1
- layer_prune_epoch = (2 if sparsity_ratio > 0 else 1) if layer_prune_ratio > 0 else -1
-
- for epoch in range(args.epochs):
- if epoch == weight_prune_epoch and prune_mask is None:
- print(f"\nGroup-wise sparsification: {sparsity_ratio*100:.0f}% of blocks (group_size={_GROUP_SIZE})...")
- pruned_params = _block_prune_params(jax_utils.unreplicate(ema_params), prune_ratio=sparsity_ratio, group_size=_GROUP_SIZE)
+ prune_interval = getattr(args, "prune_interval", 100)
+ prune_start_frac = getattr(args, "prune_start_frac", 0.33)
+ prune_end_frac = getattr(args, "prune_end_frac", 0.67)
- pruned_np = jax.tree.map(np.array, pruned_params)
- total_p = sum(x.size for x in jax.tree.leaves(pruned_np))
- zero_p = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(pruned_np))
- print(f" Post-prune sparsity: {zero_p/total_p*100:.2f}% ({zero_p:,}/{total_p:,} near-zero)")
+ weight_prune_epoch = 0 if sparsity_ratio > 0 else -1
+ layer_prune_epoch = (1 if sparsity_ratio > 0 else 0) if layer_prune_ratio > 0 else -1
- val_causal = make_causal_mask(args.max_dec_len)
- total_loss, total_toks = 0.0, 0.0
- for vb in get_batches(val_enc, val_dec_in, val_dec_tgt, args.batch_size, shuffle=False):
- vl, vt = val_loss_fn(pruned_params, vb[0], vb[1], vb[2], val_causal)
- total_loss += float(vl)
- total_toks += float(vt)
- prune_val_ppl = float(math.exp(total_loss / max(total_toks, 1)))
- print(f" Post-prune val ppl: {prune_val_ppl:.2f} (was {last_val_ppl:.2f})" if last_val_ppl is not None else f" Post-prune val ppl: {prune_val_ppl:.2f}")
-
- state = jax_utils.unreplicate(state)
- state = state.replace(params=pruned_params)
- ema_params = jax.tree.map(jnp.copy, pruned_params)
- prune_mask = jax.tree.map(lambda w: (jnp.abs(w) > 1e-8).astype(jnp.float32), pruned_params)
- prune_mask = jax_utils.replicate(prune_mask)
- state = jax_utils.replicate(state)
- ema_params = jax_utils.replicate(ema_params)
- print(f"\nContinuing training with sparsity mask locked...\n")
+ for epoch in range(args.epochs):
+ if epoch == weight_prune_epoch and not gradual_pruning_done:
+ t_start = int(num_batches * prune_start_frac)
+ t_end = int(num_batches * prune_end_frac)
+ print(f"\nGradual magnitude pruning: 0% -> {sparsity_ratio*100:.0f}% over epoch {epoch+1} "
+ f"(steps {t_start}-{t_end}/{num_batches}, interval={prune_interval}, group_size={_GROUP_SIZE})")
+ epoch_step = 0
if epoch == layer_prune_epoch:
print(f"\nLayer pruning: removing {layer_prune_ratio*100:.0f}% of layers...")
@@ -609,6 +644,15 @@ def train(args):
causal_mask,
)
+ # --- Gradual magnitude pruning (epoch 1 only) ---
+ if epoch == weight_prune_epoch and not gradual_pruning_done:
+ epoch_step += 1
+ current_sparsity = _cubic_sparsity_schedule(epoch_step, t_start, t_end, sparsity_ratio)
+ if epoch_step >= t_start and epoch_step % prune_interval == 0 and current_sparsity > 0:
+ ema_unr = jax_utils.unreplicate(ema_params)
+ _, mask = _make_prune_mask(ema_unr, current_sparsity, _GROUP_SIZE)
+ prune_mask = jax_utils.replicate(mask)
+
if prune_mask is not None:
state = state.replace(params=jax.tree.map(lambda w, m: w * m, state.params, prune_mask))
ema_params = jax.tree.map(lambda w, m: w * m, ema_params, prune_mask)
@@ -628,7 +672,12 @@ def train(args):
total_toks += float(vt)
del _eval_params
last_val_ppl = float(math.exp(total_loss / max(total_toks, 1)))
- pbar.set_postfix(loss=f"{loss_val:.4f}", val_ppl=f"{last_val_ppl:.2f}" if last_val_ppl is not None else "?")
+
+ # --- tqdm postfix with sparsity during epoch 1 ---
+ postfix = {"loss": f"{loss_val:.4f}", "val_ppl": f"{last_val_ppl:.2f}" if last_val_ppl is not None else "?"}
+ if epoch == weight_prune_epoch and not gradual_pruning_done:
+ postfix["sparsity"] = f"{current_sparsity*100:.1f}%"
+ pbar.set_postfix(**postfix)
if use_wandb:
log_dict = {
@@ -639,10 +688,29 @@ def train(args):
"train/tokens_per_sec": tokens_per_batch / dt,
"train/step": global_step,
}
+ if epoch == weight_prune_epoch and not gradual_pruning_done:
+ log_dict["train/scheduled_sparsity"] = current_sparsity
if global_step % eval_every == 0 or global_step == total_steps:
log_dict["val/ppl"] = last_val_ppl
wandb.log(log_dict)
+ # --- Lock mask after epoch 1 gradual pruning ---
+ if epoch == weight_prune_epoch and not gradual_pruning_done:
+ gradual_pruning_done = True
+ if prune_mask is None:
+ # Edge case: epoch had fewer steps than t_start
+ ema_unr = jax_utils.unreplicate(ema_params)
+ _, mask = _make_prune_mask(ema_unr, sparsity_ratio, _GROUP_SIZE)
+ prune_mask = jax_utils.replicate(mask)
+ state = state.replace(params=jax.tree.map(lambda w, m: w * m, state.params, prune_mask))
+ ema_params = jax.tree.map(lambda w, m: w * m, ema_params, prune_mask)
+ final_pruned = jax.tree.map(np.array, jax_utils.unreplicate(ema_params))
+ total_p = sum(x.size for x in jax.tree.leaves(final_pruned))
+ zero_p = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(final_pruned))
+ print(f"\n Gradual pruning complete — mask locked.")
+ print(f" Final sparsity: {zero_p/total_p*100:.2f}% ({zero_p:,}/{total_p:,} near-zero)")
+ del final_pruned
+
epoch_avg_loss = sum(losses) / len(losses) if losses else float("nan")
final_loss = losses[-1] if losses else float("nan")
final_ppl = math.exp(final_loss) if not math.isnan(final_loss) else float("nan")
@@ -753,44 +821,3 @@ def train(args):
print("\nTraining complete.")
-def sweep(args):
- import wandb
- import yaml
-
- with open(args.sweep_config) as f:
- sweep_config = yaml.safe_load(f)
-
- def sweep_train():
- wandb.init(project=args.project)
- for key, value in dict(wandb.config).items():
- if hasattr(args, key):
- setattr(args, key, value)
- args.wandb = True
- train(args)
- wandb.finish()
-
- sweep_id = wandb.sweep(sweep_config, project=args.project)
- wandb.agent(sweep_id, function=sweep_train, count=args.count)
-
-
-def parse_args():
- parser = argparse.ArgumentParser(description="Train encoder-decoder transformer on TinyStories")
- parser.add_argument("--epochs", type=int, default=2)
- parser.add_argument("--batch-size", type=int, default=32)
- parser.add_argument("--lr", type=float, default=3e-4)
- parser.add_argument("--muon-lr", type=float, default=0.02)
- parser.add_argument("--d-model", type=int, default=128)
- parser.add_argument("--num-heads", type=int, default=4)
- parser.add_argument("--num-layers", type=int, default=2)
- parser.add_argument("--max-enc-len", type=int, default=128)
- parser.add_argument("--max-dec-len", type=int, default=128)
- parser.add_argument("--max-samples", type=int, default=10000)
- parser.add_argument("--log-every", type=int, default=50)
- parser.add_argument("--warmup-ratio", type=float, default=0.05)
- parser.add_argument("--checkpoint-dir", type=str, default="checkpoints")
- parser.add_argument("--seed", type=int, default=42)
- return parser.parse_args()
-
-
-if __name__ == "__main__":
- train(parse_args())
diff --git a/sweep.yaml b/sweep.yaml
deleted file mode 100644
index 6bd0806..0000000
--- a/sweep.yaml
+++ /dev/null
@@ -1,27 +0,0 @@
-method: bayes
-metric:
- name: epoch/avg_loss
- goal: minimize
-parameters:
- lr:
- distribution: log_uniform_values
- min: 1.0e-5
- max: 1.0e-2
- muon_lr:
- distribution: log_uniform_values
- min: 5.0e-3
- max: 0.1
- warmup_ratio:
- distribution: uniform
- min: 0.01
- max: 0.15
- dropout:
- values: [0.0, 0.05, 0.1, 0.15, 0.2]
- batch_size:
- values: [16, 32, 64]
- d_model:
- values: [64, 128, 256]
- num_layers:
- values: [2, 3, 4]
- num_heads:
- values: [2, 4, 8]