Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
Add experiments
HenryNdubuaku126117fafdd2
diff --git a/README.md b/README.mdindex 1a124f6..ca53428 100644--- a/README.md+++ b/README.md@@ -160,23 +160,23 @@ needle [command]~27 GB (from ~4.5 TB full dataset)──────────────────────────────────────────- 300M multimodal training estimates+ 1B multimodal training estimates──────────────────────────────────────────- Total FLOPs 6 × 300M × 100B = 1.8e20- MFU (300M model) ~15% → ~138 effective TFLOPS/chip+ Total FLOPs 6 × 1B × 100B = 6e20+ MFU (1B model) ~30% → ~275 effective TFLOPS/chipAudio overhead <1% of total (negligible)- ┌────────────────────┬──────────┬──────────┐- │ │ v6e-8 │ v6e-16 │- ├────────────────────┼──────────┼──────────┤- │ Chips │ 8 │ 16 │- │ Total HBM │ 256 GB │ 512 GB │- │ Scaling eff. │ 0.9× │ 0.8× │- │ Eff. TFLOPS │ 992 │ 1,766 │- │ Est. time │ ~50h │ ~29h │- │ On-demand $/hr │ $21.60 │ $43.20 │- │ Est. total cost │ ~$1,080 │ ~$1,250 │- └────────────────────┴──────────┴──────────┘+ ┌────────────────────┬──────────┬──────────┬──────────┐+ │ │ v6e-8 │ v6e-16 │ v6e-32 │+ ├────────────────────┼──────────┼──────────┼──────────┤+ │ Chips │ 8 │ 16 │ 32 │+ │ Total HBM │ 256 GB │ 512 GB │ 1024 GB │+ │ Scaling eff. │ 0.9× │ 0.8× │ 0.7× │+ │ Eff. TFLOPS │ 1,983 │ 3,525 │ 6,160 │+ │ Est. time │ ~84h │ ~47h │ ~27h │+ │ On-demand $/hr │ $21.60 │ $43.20 │ $86.40 │+ │ Est. total cost │ ~$1,810 │ ~$2,030 │ ~$2,330 │+ └────────────────────┴──────────┴──────────┴──────────┘```## Setup For TPU/GCPdiff --git a/experiements/architecture.txt b/experiements/architecture.txtindex e69de29..50965f5 100644--- a/experiements/architecture.txt+++ b/experiements/architecture.txt@@ -0,0 +1,390 @@+================================================================================+Assignee: Noah Cylich & Justin Lee+Deadline: 7th March+================================================================================++Thesis: Three complementary compression axes; Matryoshka dimension pruning,+group-aligned block sparsity, and INT4 quantization-aware training. can be+jointly applied during training to produce a single checkpoint that is+deployable at multiple accuracy-efficiency operating points. Training once at+d_model=D with MRL loss at dimensions {D, D/2, D/4, D/8, ...} yields a family+of models from one training run, each further compressible via sparsification.++================================================================================+SECTION 1 — EXPERIMENTAL AXES+================================================================================++Independent variables:+ A. MRL deployment dim d' ∈ {1024, 512, 256, 128, 64}+ B. Sparsity ratio s ∈ {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}+ C. Group size g ∈ {16, 32, 64, 128}+ D. FFN activation ∈ {dReLU, SwiGLU, GeGLU}+ E. Model scale ∈ {TOY-128, MAIN-1024}+ F. QAT ∈ {off, int4-group32}++Dependent variables (measured at EACH MRL dimension):+ - Validation perplexity (TinyStories)+ - Quantized validation perplexity (INT4)+ - WikiText-2 perplexity+ - LAMBADA accuracy+ - HellaSwag accuracy+ - ARC-Easy accuracy+ - Estimated parameter count at deployment dim+ - Throughput (tokens/sec on inference)+ - Bigram repetition rate (generation diversity)+ - Block sparsity (% of fully-zero groups, group_size=g)+ - Effective FLOPs (est. params × density)++================================================================================+SECTION 2 — MODEL CONFIGURATIONS+================================================================================++Architecture: MemoryMixer encoder + Transformer decoder, shared embedding,+GQA (grouped query attention), ZC-RMSNorm, RoPE, SentencePiece BPE (8192).++TOY-128 (rapid iteration):+ mrl_dims=[128, 64], d_model=128, num_heads=4, num_kv_heads=2,+ num_enc_layers=2, num_dec_layers=2, d_ff=512,+ max_enc_len=128, max_dec_len=128, vocab=8192, dtype=bfloat16+ needle train --toy --seed {42,137,256}++MAIN-1024 (paper results):+ mrl_dims=[1024, 512, 256, 128, 64], d_model=1024, num_heads=4,+ num_kv_heads=2, num_enc_layers=4, num_dec_layers=2, d_ff=4096,+ max_enc_len=256, max_dec_len=256, vocab=8192, dtype=bfloat16+ needle train --seed {42,137,256}++Both trained on TinyStories (1M samples, encoder-decoder split at 30%).+Optimizer: Muon (kernels) + AdamW (embeddings/norms), WSD schedule.+All experiments report metrics at each MRL dimension slice.++================================================================================+SECTION 3 — EXPERIMENT GROUPS+================================================================================++All experiments use 3 seeds (42, 137, 256) for statistical significance.+Report mean +/- std on all metrics. Every run automatically reports val PPL+and INT4-quantized val PPL at each MRL dimension slice.++-------------------------------------------------------------------------------+GROUP 1: Dense Baselines + MRL Dimension Scaling (Table 1, Figure 1)+-------------------------------------------------------------------------------+Purpose: Establish quality at each deployment dimension from a single training+run. This is the MRL baseline — no sparsification.++ 1a. MAIN-1024, 3 epochs, no pruning, MRL dims [1024,512,256,128,64]+ needle train --sparsity-ratio 0.0 --seed {42,137,256}++ 1b. TOY-128, 3 epochs, no pruning, MRL dims [128,64]+ needle train --toy --sparsity-ratio 0.0 --seed {42,137,256}++Deliverables:+ - Table: val PPL and quant PPL at each MRL dim (1024, 512, 256, 128, 64)+ - Line plot: deployment dim vs val PPL — the MRL scaling curve (Figure 1a)+ - Line plot: deployment dim vs estimated param count (Figure 1b)+ - Benchmark scores at each dim+ - These are the denominators for all relative comparisons++-------------------------------------------------------------------------------+GROUP 2: MRL + Group-Wise Sparsification (Table 2, Figure 2)+ *** CORE CONTRIBUTION ***+-------------------------------------------------------------------------------+Purpose: Characterise how MRL dimension scaling interacts with block sparsity.+At smaller deployment dims, does sparsification hurt more or less?++Schedule: Epoch 1 dense, epochs 2-3 with sparsity mask locked.++ For s in {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}:+ 2a. MAIN-1024, 3 epochs, --sparsity-ratio s+ needle train --sparsity-ratio {s} --seed {42,137,256}++ 2b. Select best 2 sparsity ratios from 2a, run with 4 epochs (recovery)+ needle train --epochs 4 --sparsity-ratio {s} --seed {42,137,256}++Deliverables:+ - 2D heatmap: (sparsity, MRL dim) → val PPL (Figure 2a)+ - Pareto curve per MRL dim: sparsity % vs val PPL (Figure 2b)+ - Table: all metrics at each (sparsity, dim) pair+ - Compression spectrum: for best s, show the range of models:+ d'=1024 (full) → d'=64 (smallest) with estimated params and PPL+ - Analysis: does sparsification degrade smaller MRL slices more?+ (Hypothesis: smaller slices are more sensitive because fewer dimensions+ carry the representation — each zeroed weight matters more)+ - Analysis: multiplicative compression — MRL dimension reduction ×+ sparsity = total compression ratio++-------------------------------------------------------------------------------+GROUP 3: Recovery Epoch Scaling (Table 3, Figure 3)+-------------------------------------------------------------------------------+Purpose: How many recovery epochs are needed after sparsification?++Pick the best sparsity ratio from Group 2a. Vary total epochs:++ For E in {2, 3, 4, 5, 6}:+ 3a. MAIN-1024, E epochs, --sparsity-ratio {s}+ (sparsification at epoch 1; recovery = E - 1 epochs)++Deliverables:+ - Line plot per MRL dim: recovery epochs vs val PPL (Figure 3)+ - Analysis: do smaller MRL dims need more recovery?++-------------------------------------------------------------------------------+GROUP 4: QAT Interaction Study (Table 4, Figure 4)+-------------------------------------------------------------------------------+Purpose: Show that group-wise sparsification is compatible with INT4 QAT+because both operate on the same group_size=32 granularity. Measure at+each MRL dimension — INT4 quantization error at smaller dims may compound.++ 4a. Dense + QAT (baseline, = Group 1a)+ 4b. Sparsified (s=0.3) + QAT+ 4c. Sparsified (s=0.6) + QAT++ For comparison, also run 4b-4c WITHOUT QAT by modifying _train_step+ to not call _quantize_params.++Deliverables:+ - Table: (QAT on/off) × sparsity × MRL dim → val PPL+ - Analysis: does QAT degrade more at smaller MRL dims?+ (Hypothesis: yes — fewer dimensions means each quantization error+ has proportionally larger impact on output quality)+ - Effective compression: MRL dim reduction × sparsity × INT4 → total+ bits per effective parameter++-------------------------------------------------------------------------------+GROUP 5: Group Size Ablation (Table 5, Figure 5)+-------------------------------------------------------------------------------+Purpose: Determine optimal block granularity for sparsification.++Fixed: MAIN-1024, s=0.3, 3 epochs, QAT on.++ For g in {16, 32, 64, 128}:+ 5a. needle train --sparsity-ratio 0.3 --group-size {g} \+ --seed {42,137,256}++ High sparsity stress test:+ For g in {16, 32, 64, 128}:+ 5b. needle train --sparsity-ratio 0.6 --group-size {g} \+ --seed {42,137,256}++Deliverables:+ - Line plot per MRL dim: group size vs val PPL (Figure 5a)+ - Table: all metrics at each (group size, MRL dim)+ - Effective bits-per-param: bpp = (1 - sparsity) * (4 + 32/g)+ - Analysis: does optimal group size change with MRL deployment dim?++-------------------------------------------------------------------------------+GROUP 6: Activation Function x Sparsification Interaction (Table 6, Figure 6)+ *** KEY FINDING ***+-------------------------------------------------------------------------------+Purpose: Test whether activation function choice determines sparsification+robustness across MRL dimension slices.++Activation definitions:+ - dReLU: ReLU(gate) * ReLU(up) — naturally ~75% sparse activations+ - SwiGLU: SiLU(gate) * up — smooth, dense activations+ - GeGLU: GELU(gate) * up — smooth, nearly dense activations++ 6a. Dense baselines (no sparsification) — isolate activation quality+ For act in {drelu, swiglu, geglu}:+ needle train --activation {act} --sparsity-ratio 0.0 \+ --seed {42,137,256}++ 6b. Sparsification — test robustness per activation+ For act in {drelu, swiglu, geglu}, s in {0.3, 0.6}:+ needle train --activation {act} --sparsity-ratio {s} \+ --seed {42,137,256}++Deliverables:+ - Grouped bar chart per MRL dim: activation × compression → val PPL (Fig 6a)+ - Relative degradation: (sparse_ppl / dense_ppl) per activation per dim+ (Figure 6b) — KEY CHART: sparsification robustness not just raw quality+ - Analysis: does dReLU's natural activation sparsity help or hurt at+ smaller MRL dims where redundancy is already low?++-------------------------------------------------------------------------------+GROUP 7: MRL Dimension-Specific Analysis (Table 7, Figure 7)+ *** MRL-SPECIFIC CONTRIBUTION ***+-------------------------------------------------------------------------------+Purpose: Deep-dive into what the model learns at each MRL dimension slice.++Using best checkpoint from Group 2a:++ 7a. Representation similarity across dims+ - CKA (centered kernel alignment) between d'=1024 and each smaller d'+ - Measure: do smaller dims learn a subspace of the full representation?++ 7b. Per-dim generation quality+ - Generate from each MRL dim slice at inference time+ - Compare generation diversity, coherence, repetition rate++ 7c. MRL loss weighting ablation+ - Uniform weighting (current): L = (1/K) * sum(L_k)+ - Scaled weighting: L = sum(d_k/D * L_k) (larger dims weighted more)+ - Inverse weighting: L = sum(D/d_k * L_k) (smaller dims weighted more)++Deliverables:+ - CKA heatmap: dim × dim similarity (Figure 7a)+ - Table: generation quality metrics at each dim+ - Ablation table: MRL loss weighting strategy × dim → val PPL++-------------------------------------------------------------------------------+GROUP 8: Inference Efficiency Validation (Table 8, Figure 8)+-------------------------------------------------------------------------------+Purpose: Translate theoretical compression into real speedup across the+full spectrum of deployment options.++For each Pareto-optimal checkpoint from Groups 1-2, at each MRL dim:+ needle test --checkpoint {path} --throughput-runs 50++Measure at each deployment dim:+ - Tokens/sec (decode throughput)+ - Latency per generation (ms)+ - Peak memory (via JAX profiler)++Deliverables:+ - Scatter plot: effective params vs actual throughput (Figure 8a)+ - Table: all Pareto-optimal (s, d') with throughput + quality+ - The "menu" figure: one chart showing ALL deployment options from a+ single checkpoint, from largest (d'=1024, no pruning) to smallest+ (d'=64, s=0.80) — THIS IS THE PAPER'S HERO FIGURE++================================================================================+SECTION 4 — EVALUATION PROTOCOL+================================================================================++For every checkpoint, evaluate at EACH MRL dimension slice:++ 1. Validation perplexity (TinyStories val split)+ - Full-precision val PPL+ - INT4-quantized val PPL (at each MRL dim)+ needle test --checkpoint {path} --max-eval-samples 5000++ 2. Benchmark suite (at full dim and smallest viable dim)+ needle evaluate --checkpoint {path} \+ --benchmarks wikitext2 lambada hellaswag arc_easy --max-samples 500++ 3. Generation quality at each MRL dim+ needle test --checkpoint {path}+ Report: bigram repetition rate, avg generation length, sample outputs++ 4. Throughput at each MRL dim+ needle test --checkpoint {path} --throughput-runs 50++================================================================================+SECTION 5 — EXPECTED TABLES AND FIGURES+================================================================================++Tables:+ T1. Dense baselines + MRL scaling curve (val PPL at each dim)+ T2. MRL + sparsification (sparsity × dim → val PPL)+ T3. Recovery epoch scaling (epochs × MRL dims)+ T4. QAT interaction (sparsity × {on,off} × MRL dims)+ T5. Group size ablation (4 sizes × MRL dims)+ T6. Activation × sparsification interaction (3 act × compression × MRL dims)+ T7. MRL dimension analysis (CKA, generation quality, loss weighting)+ T8. Inference efficiency (Pareto-optimal checkpoints × MRL dims)++Figures:+ F1. MRL scaling curve: deployment dim vs PPL + param count+ F2. MRL + sparsity: (a) heatmap, (b) Pareto per dim — KEY FIGURE+ F3. Recovery epoch scaling per MRL dim+ F4. QAT interaction: sparsity × QAT × MRL dim+ F5. Group size: (a) g vs PPL per dim, (b) effective bpp+ F6. Activation × sparsification: (a) grouped bars per dim, (b) degradation+ F7. MRL representation analysis: (a) CKA heatmap, (b) gen quality vs dim+ F8. Hero figure: full deployment menu from single checkpoint++================================================================================+SECTION 6 — COMPUTE BUDGET+================================================================================++Per-run cost (MAIN-1024, 3 epochs, 1M samples, 4x TPU v5 lite):+ ~15 minutes training + ~5 minutes MRL evaluation (5 dims × val + quant)++Estimated total runs:+ Group 1: 2 configs × 3 seeds = 6 runs+ Group 2: 8 ratios × 3 seeds + 2 ratios × 3 seeds (rec)= 30 runs+ Group 3: 5 epoch counts × 3 seeds = 15 runs+ Group 4: 3 methods × 2 (QAT on/off) × 3 seeds = 18 runs+ Group 5: 4 sizes × 2 sparsity × 3 seeds = 24 runs+ Group 6: 3 act × (3 + 6) × 3 seeds = 27 runs+ Group 7: 3 weighting strategies × 3 seeds = 9 runs+ Group 8: evaluation only (reuses checkpoints) = 0 runs+ ────────────────────────────────────────────────────────────────────+ Total: 129 runs++At ~20 min/run: ~43 TPU-hours (manageable on v5litepod-4)++================================================================================+SECTION 7 — ABLATION CHECKLIST+================================================================================++Before submission, verify these are addressed:++ [ ] Statistical significance: 3 seeds, report mean +/- std+ [ ] MRL scaling: smooth degradation curve from full dim to smallest+ [ ] MRL + sparsity composability: both axes combine gracefully+ [ ] Matched comparisons: compare methods at SAME effective param count+ [ ] Recovery saturation: diminishing returns on fine-tuning epochs+ [ ] QAT compatibility: sparsity + quantization + MRL are not antagonistic+ [ ] Generation quality at each MRL dim, not just perplexity+ [ ] Group size sensitivity: optimal g, interaction with MRL dim+ [ ] Activation × sparsification × MRL dim: three-way interaction+ [ ] MRL representation analysis: CKA, subspace structure+ [ ] Honest limitations: sparse matmuls don't accelerate on current JAX;+ MRL dimension reduction requires custom kernel for real speedup+ [ ] Comparison to baselines: training K separate small models vs one MRL++================================================================================+SECTION 8 — EXECUTION ORDER+================================================================================++Phase 1 (Day 1): Groups 1 + 2 (36 runs, ~12 hrs)+ Baselines and sparsification sweeps, all with MRL evaluation.+ Deliverable: MRL scaling curves and sparsity Pareto frontiers.++Phase 2 (Day 2): Groups 3 + 4 (33 runs, ~11 hrs)+ Recovery and QAT ablations.+ Deliverable: complete ablation tables.++Phase 3 (Day 3): Groups 5 + 6 (51 runs, ~17 hrs)+ Group size and activation interaction studies.+ Deliverable: granularity and activation robustness analysis.++Phase 4 (Day 4): Groups 7 + 8 + writing (9 runs + eval)+ MRL analysis, inference benchmarks, paper draft.+ Deliverable: hero figure, all tables populated.++================================================================================+SECTION 9 — PAPER NARRATIVE+================================================================================++Framing: "Train once, deploy anywhere."++A single training run with Matryoshka loss produces a checkpoint that can be+deployed at any dimension slice. Each slice can be further compressed with+group-wise sparsification. The result is a deployment MENU:++ ┌─────────────────────────────────────────────────────────────────────┐+ │ Deployment option Est. params Val PPL Quant PPL │+ │ ───────────────────────────────────────────────────────────────── │+ │ d'=1024, full ~44M X.XX X.XX │+ │ d'=512, full ~11M X.XX X.XX │+ │ d'=512, s=0.3 ~7M X.XX X.XX │+ │ d'=512, s=0.6 ~4M X.XX X.XX │+ │ d'=256, full ~3M X.XX X.XX │+ │ d'=256, s=0.3 ~2M X.XX X.XX │+ │ d'=128, full ~0.8M X.XX X.XX │+ │ d'=128, s=0.5 ~0.4M X.XX X.XX │+ │ d'=64, full ~0.2M X.XX X.XX │+ │ d'=64, s=0.8 ~0.04M X.XX X.XX │+ └─────────────────────────────────────────────────────────────────────┘++Key claims:+ 1. MRL dimension pruning is nearly free — minimal degradation at 2x reduction+ 2. Sparsification composes multiplicatively with MRL+ 3. A single checkpoint spans 100x+ parameter range+ 4. INT4 QAT is compatible with both compression axes+ 5. MemoryMixer encoder is more compressible than standard transformer encoder++================================================================================diff --git a/experiements/datasets.txt b/experiements/datasets.txtindex e69de29..555dc0c 100644--- a/experiements/datasets.txt+++ b/experiements/datasets.txt@@ -0,0 +1,26 @@+================================================================================++Task: Design & gather multimodal dataset, text tokenization, speech processing, and tool call strategy.+Deliverable: A pipleine of preprocessed dataset, from which the codebase can stream datasets from.+Assignee: Parkirat Sandhu & Jakub Mroz+Deadline: 7th March++================================================================================++Text dataset: SYNTH (https://huggingface.co/datasets/PleIAs/SYNTH)+Transcribe datasets: Emilia-Large, SPGI Speech+Text tool call datasets: xlam-function-calling-60k, xlam-irrelevance-7.5k, Salesforce/APIGen-MT-5k, APIGen-Irrelevant-5k, glaive-function-calling, Team-ACE/ToolACE+Voice tool call datasets: Use a good TTS model to generate voice commands from the tool call dataset prompts++================================================================================++Note:+1. LEAP makes sense as we wish the dataset to preprocess and pretokenize the inputs before injesting into the model.+2. We are training a cutom sentencepiece tokenizer with a small vocab size, given how constrained the decode is.+3. Requires solid cloud infra experience, latency in injestion will slow down training.+4. Record all obersvations, they will go into the paper!+5. Design the batching (mix of speech and text and tools) experimantally.+6. We should have a <transcribe> and <tools> tag for the two tasks we train on.++================================================================================+diff --git a/experiements/pruning.txt b/experiements/pruning.txtdeleted file mode 100644index 59828d7..0000000--- a/experiements/pruning.txt+++ /dev/null@@ -1,492 +0,0 @@-================================================================================-MULTI-AXIS STRUCTURED COMPRESSION OF SMALL ENCODER-DECODER TRANSFORMERS:-MATRYOSHKA DIMENSIONS, BLOCK SPARSITY, LAYER PRUNING, AND INT4 QAT-================================================================================-Target venue: ICLR 2026--Thesis: Four complementary compression axes — Matryoshka dimension pruning,-group-aligned block sparsity, layer removal, and INT4 quantization-aware-training — can be jointly applied during training in a staged schedule to-produce a single checkpoint that is deployable at multiple accuracy-efficiency-operating points. Training once at d_model=D with MRL loss at dimensions-{D, D/2, D/4, D/8, ...} yields a family of models from one training run,-each further compressible via sparsification and layer pruning.--================================================================================-SECTION 1 — EXPERIMENTAL AXES-================================================================================--Independent variables:- A. MRL deployment dim d' ∈ {1024, 512, 256, 128, 64}- B. Sparsity ratio s ∈ {0.0, 0.10, 0.25, 0.33, 0.50}- C. Layer prune ratio l ∈ {0.0, 0.125, 0.25, 0.375, 0.50}- D. Group size g ∈ {16, 32, 64, 128}- E. Pruning schedule ∈ {staged, simultaneous, reverse-order}- F. FFN activation ∈ {dReLU, SwiGLU, GeGLU}- G. Model scale ∈ {TOY-128, MAIN-1024}- H. QAT ∈ {off, int4-group32}--Dependent variables (measured at EACH MRL dimension):- - Validation perplexity (TinyStories)- - Quantized validation perplexity (INT4)- - WikiText-2 perplexity- - LAMBADA accuracy- - HellaSwag accuracy- - ARC-Easy accuracy- - Estimated parameter count at deployment dim- - Throughput (tokens/sec on inference)- - Bigram repetition rate (generation diversity)- - Block sparsity (% of fully-zero groups, group_size=g)- - Effective FLOPs (est. params × density)--================================================================================-SECTION 2 — MODEL CONFIGURATIONS-================================================================================--Architecture: MemoryMixer encoder + Transformer decoder, shared embedding,-GQA (grouped query attention), ZC-RMSNorm, RoPE, SentencePiece BPE (8192).--TOY-128 (rapid iteration):- mrl_dims=[128, 64], d_model=128, num_heads=4, num_kv_heads=2,- num_enc_layers=2, num_dec_layers=2, d_ff=512,- max_enc_len=128, max_dec_len=128, vocab=8192, dtype=bfloat16- needle train --toy --seed {42,137,256}--MAIN-1024 (paper results):- mrl_dims=[1024, 512, 256, 128, 64], d_model=1024, num_heads=4,- num_kv_heads=2, num_enc_layers=4, num_dec_layers=2, d_ff=4096,- max_enc_len=256, max_dec_len=256, vocab=8192, dtype=bfloat16- needle train --seed {42,137,256}--Both trained on TinyStories (1M samples, encoder-decoder split at 30%).-Optimizer: Muon (kernels) + AdamW (embeddings/norms), WSD schedule.-All experiments report metrics at each MRL dimension slice.--================================================================================-SECTION 3 — EXPERIMENT GROUPS-================================================================================--All experiments use 3 seeds (42, 137, 256) for statistical significance.-Report mean +/- std on all metrics. Every run automatically reports val PPL-and INT4-quantized val PPL at each MRL dimension slice.----------------------------------------------------------------------------------GROUP 1: Dense Baselines + MRL Dimension Scaling (Table 1, Figure 1)---------------------------------------------------------------------------------Purpose: Establish quality at each deployment dimension from a single training-run. This is the MRL baseline — no sparsification or layer pruning.-- 1a. MAIN-1024, 3 epochs, no pruning, MRL dims [1024,512,256,128,64]- needle train --sparsity-ratio 0.0 --layer-prune-ratio 0.0 \- --seed {42,137,256}-- 1b. TOY-128, 3 epochs, no pruning, MRL dims [128,64]- needle train --toy --sparsity-ratio 0.0 --layer-prune-ratio 0.0 \- --seed {42,137,256}--Deliverables:- - Table: val PPL and quant PPL at each MRL dim (1024, 512, 256, 128, 64)- - Line plot: deployment dim vs val PPL — the MRL scaling curve (Figure 1a)- - Line plot: deployment dim vs estimated param count (Figure 1b)- - Benchmark scores at each dim- - These are the denominators for all relative comparisons----------------------------------------------------------------------------------GROUP 2: MRL + Group-Wise Sparsification (Table 2, Figure 2)---------------------------------------------------------------------------------Purpose: Characterise how MRL dimension scaling interacts with block sparsity.-At smaller deployment dims, does sparsification hurt more or less?--Schedule: Epoch 1 dense, epochs 2-3 with sparsity mask locked.-- For s in {0.10, 0.25, 0.33, 0.50}:- 2a. MAIN-1024, 3 epochs, --sparsity-ratio s- needle train --sparsity-ratio {s} --layer-prune-ratio 0.0 \- --seed {42,137,256}--Deliverables:- - 2D heatmap: (sparsity, MRL dim) → val PPL (Figure 2a)- - Pareto curve per MRL dim: sparsity % vs val PPL (Figure 2b)- - Table: all metrics at each (sparsity, dim) pair- - Analysis: does sparsification degrade smaller MRL slices more?- (Hypothesis: smaller slices are more sensitive because fewer dimensions- carry the representation — each zeroed weight matters more)----------------------------------------------------------------------------------GROUP 3: MRL + Layer Pruning (Table 3, Figure 3)---------------------------------------------------------------------------------Purpose: Characterise how MRL dimension scaling interacts with layer removal.--Schedule: Epoch 1 dense, epochs 2-3 with reduced architecture.-- For l in {0.125, 0.25, 0.375, 0.50}:- 3a. MAIN-1024 (4 enc + 2 dec layers), 3 epochs, --layer-prune-ratio l- needle train --sparsity-ratio 0.0 --layer-prune-ratio {l} \- --seed {42,137,256}- Expected surviving layers: 5, 4-5, 4, 3 (of 6 total)--Deliverables:- - 2D heatmap: (layer ratio, MRL dim) → val PPL (Figure 3a)- - Pareto curve: param count vs val PPL at each MRL dim (Figure 3b)- - Per-run encoder/decoder surviving layer counts- - Analysis: does the pruner prefer removing encoder or decoder layers?- (Hypothesis: encoder MemoryMixer blocks are pruned more aggressively- since the decoder carries more of the generation burden)----------------------------------------------------------------------------------GROUP 4: Full Pipeline — MRL + Sparsification + Layer Pruning (Table 4, Fig 4)- *** CORE CONTRIBUTION ***---------------------------------------------------------------------------------Purpose: Show that all three compression axes compose — a single checkpoint-trained with MRL loss, then sparsified and layer-pruned, produces a family-of models spanning 10-100x parameter range. This is the central claim.--Schedule (staged):- Epoch 1: dense training with MRL loss- Epoch 2: apply group-wise sparsification, train with mask + MRL- Epoch 3: apply layer pruning, train reduced architecture + mask + MRL- (Optional epoch 4: recovery fine-tuning)-- For (s, l) in grid:- {(0.10, 0.125), (0.10, 0.25),- (0.25, 0.125), (0.25, 0.25), (0.25, 0.375),- (0.33, 0.125), (0.33, 0.25),- (0.50, 0.25), (0.50, 0.50)}-- 4a. MAIN-1024, 3 epochs- needle train --sparsity-ratio {s} --layer-prune-ratio {l} \- --seed {42,137,256}-- 4b. Select best 4 (s,l) pairs from 4a, run with 4 epochs (recovery)- needle train --epochs 4 --sparsity-ratio {s} --layer-prune-ratio {l} \- --seed {42,137,256}--Deliverables:- - 3D surface: (s, l, MRL dim) → val PPL (Figure 4a, sliced by dim)- - Pareto overlay at each MRL dim: full-pipeline vs sparsity-only vs- layer-only at matched effective params (Figure 4b) — KEY FIGURE- - Table: full metrics for all (s, l) pairs at each MRL dim- - Compression spectrum: for best (s, l), show the range of models:- d'=1024 (full) → d'=64 (smallest) with estimated params and PPL- - Analysis: multiplicative compression — MRL dimension reduction ×- sparsity × layer removal = total compression ratio----------------------------------------------------------------------------------GROUP 5: Schedule Ablation (Table 5)---------------------------------------------------------------------------------Purpose: Justify the staged schedule (sparsify first, then prune layers).--Pick the best (s, l) pair from Group 4a. Run 3 schedules:-- 5a. Staged (default): sparsify epoch 2, layer prune epoch 3- needle train --sparsity-ratio {s} --layer-prune-ratio {l}-- 5b. Reverse: layer prune epoch 2, sparsify epoch 3- [Requires code change to swap epoch triggers]-- 5c. Simultaneous: both at epoch 2, recover epoch 3- [Requires code change to apply both at same epoch]--Deliverables:- - Table: val PPL at each MRL dim for each schedule- - Analysis: does schedule affect small MRL dims differently from large?----------------------------------------------------------------------------------GROUP 6: Recovery Epoch Scaling (Figure 5)---------------------------------------------------------------------------------Purpose: How many recovery epochs are needed after compression?--Pick the best (s, l) pair from Group 4a. Vary total epochs:-- For E in {2, 3, 4, 5, 6}:- 6a. MAIN-1024, E epochs, --sparsity-ratio {s} --layer-prune-ratio {l}- (pruning at epochs 2 and 3; recovery = E - 3 epochs)--Deliverables:- - Line plot per MRL dim: recovery epochs vs val PPL (Figure 5)- - Analysis: do smaller MRL dims need more recovery?----------------------------------------------------------------------------------GROUP 7: QAT Interaction Study (Table 6)---------------------------------------------------------------------------------Purpose: Show that group-wise sparsification is compatible with INT4 QAT-because both operate on the same group_size=32 granularity. Measure at-each MRL dimension — INT4 quantization error at smaller dims may compound.-- 7a. Dense + QAT (baseline, = Group 1a)- 7b. Sparsified (s=0.25) + QAT- 7c. Layer-pruned (l=0.25) + QAT- 7d. Joint (s=0.25, l=0.25) + QAT-- For comparison, also run 7b-7d WITHOUT QAT by modifying _train_step- to not call _quantize_params.--Deliverables:- - Table: (QAT on/off) × compression method × MRL dim → val PPL- - Analysis: does QAT degrade more at smaller MRL dims?- (Hypothesis: yes — fewer dimensions means each quantization error- has proportionally larger impact on output quality)- - Effective compression: MRL dim reduction × sparsity × INT4 → total- bits per effective parameter----------------------------------------------------------------------------------GROUP 8: Group Size Ablation (Table 7, Figure 6)---------------------------------------------------------------------------------Purpose: Determine optimal block granularity for sparsification.--Fixed: MAIN-1024, s=0.25, 3 epochs, QAT on.-- For g in {16, 32, 64, 128}:- 8a. Sparsification only- needle train --sparsity-ratio 0.25 --layer-prune-ratio 0.0 \- --group-size {g} --seed {42,137,256}-- 8b. Joint sparsification + layer pruning (best l from Group 4a)- needle train --sparsity-ratio 0.25 --layer-prune-ratio {l} \- --group-size {g} --seed {42,137,256}-- High sparsity stress test:- For g in {16, 32, 64, 128}:- 8c. s=0.50, sparsification only- needle train --sparsity-ratio 0.50 --layer-prune-ratio 0.0 \- --group-size {g} --seed {42,137,256}--Deliverables:- - Line plot per MRL dim: group size vs val PPL (Figure 6a)- - Table: all metrics at each (group size, MRL dim)- - Effective bits-per-param: bpp = (1 - sparsity) * (4 + 32/g)- - Analysis: does optimal group size change with MRL deployment dim?----------------------------------------------------------------------------------GROUP 9: Activation Function x Pruning Interaction (Table 8, Figure 7)- *** KEY FINDING ***---------------------------------------------------------------------------------Purpose: Test whether activation function choice determines pruning robustness-across MRL dimension slices.--Activation definitions:- - dReLU: ReLU(gate) * ReLU(up) — naturally ~75% sparse activations- - SwiGLU: SiLU(gate) * up — smooth, dense activations- - GeGLU: GELU(gate) * up — smooth, nearly dense activations-- 9a. Dense baselines (no pruning) — isolate activation quality- For act in {drelu, swiglu, geglu}:- needle train --activation {act} --sparsity-ratio 0.0 \- --layer-prune-ratio 0.0 --seed {42,137,256}-- 9b. Sparsification only — test pruning robustness per activation- For act in {drelu, swiglu, geglu}, s in {0.25, 0.50}:- needle train --activation {act} --sparsity-ratio {s} \- --layer-prune-ratio 0.0 --seed {42,137,256}-- 9c. Joint — full compression- For act in {drelu, swiglu, geglu}:- needle train --activation {act} --sparsity-ratio 0.25 \- --layer-prune-ratio 0.25 --seed {42,137,256}--Deliverables:- - Grouped bar chart per MRL dim: activation × compression → val PPL (Fig 7a)- - Relative degradation: (pruned_ppl / dense_ppl) per activation per dim- (Figure 7b) — KEY CHART: pruning robustness not just raw quality- - Analysis: does dReLU's natural activation sparsity help or hurt at- smaller MRL dims where redundancy is already low?----------------------------------------------------------------------------------GROUP 10: MRL Dimension-Specific Analysis (Table 9, Figure 8)- *** MRL-SPECIFIC CONTRIBUTION ***---------------------------------------------------------------------------------Purpose: Deep-dive into what the model learns at each MRL dimension slice.--Using best checkpoint from Group 4a:-- 10a. Representation similarity across dims- - CKA (centered kernel alignment) between d'=1024 and each smaller d'- - Measure: do smaller dims learn a subspace of the full representation?-- 10b. Per-dim generation quality- - Generate from each MRL dim slice at inference time- - Compare generation diversity, coherence, repetition rate-- 10c. MRL loss weighting ablation- - Uniform weighting (current): L = (1/K) * sum(L_k)- - Scaled weighting: L = sum(d_k/D * L_k) (larger dims weighted more)- - Inverse weighting: L = sum(D/d_k * L_k) (smaller dims weighted more)--Deliverables:- - CKA heatmap: dim × dim similarity (Figure 8a)- - Table: generation quality metrics at each dim- - Ablation table: MRL loss weighting strategy × dim → val PPL----------------------------------------------------------------------------------GROUP 11: Inference Efficiency Validation (Table 10, Figure 9)---------------------------------------------------------------------------------Purpose: Translate theoretical compression into real speedup across the-full spectrum of deployment options.--For each Pareto-optimal checkpoint from Groups 2-4, at each MRL dim:- needle test --checkpoint {path} --throughput-runs 50--Measure at each deployment dim:- - Tokens/sec (decode throughput)- - Latency per generation (ms)- - Peak memory (via JAX profiler)--Deliverables:- - Scatter plot: effective params vs actual throughput (Figure 9a)- - Table: all Pareto-optimal (s, l, d') with throughput + quality- - The "menu" figure: one chart showing ALL deployment options from a- single checkpoint, from largest (d'=1024, no pruning) to smallest- (d'=64, s=0.50, l=0.50) — THIS IS THE PAPER'S HERO FIGURE--================================================================================-SECTION 4 — EVALUATION PROTOCOL-================================================================================--For every checkpoint, evaluate at EACH MRL dimension slice:-- 1. Validation perplexity (TinyStories val split)- - Full-precision val PPL- - INT4-quantized val PPL (at each MRL dim)- needle test --checkpoint {path} --max-eval-samples 5000-- 2. Benchmark suite (at full dim and smallest viable dim)- needle evaluate --checkpoint {path} \- --benchmarks wikitext2 lambada hellaswag arc_easy --max-samples 500-- 3. Generation quality at each MRL dim- needle test --checkpoint {path}- Report: bigram repetition rate, avg generation length, sample outputs-- 4. Throughput at each MRL dim- needle test --checkpoint {path} --throughput-runs 50--================================================================================-SECTION 5 — EXPECTED TABLES AND FIGURES-================================================================================--Tables:- T1. Dense baselines + MRL scaling curve (val PPL at each dim)- T2. MRL + sparsification (sparsity × dim → val PPL)- T3. MRL + layer pruning (layer ratio × dim → val PPL)- T4. Full pipeline grid (s × l at each MRL dim)- T5. Schedule ablation (3 schedules × MRL dims)- T6. QAT interaction (4 methods × {on,off} × MRL dims)- T7. Group size ablation (4 sizes × MRL dims)- T8. Activation × pruning interaction (3 act × compression × MRL dims)- T9. MRL dimension analysis (CKA, generation quality, loss weighting)- T10. Inference efficiency (Pareto-optimal checkpoints × MRL dims)--Figures:- F1. MRL scaling curve: deployment dim vs PPL + param count- F2. MRL + sparsity: (a) heatmap, (b) Pareto per dim- F3. MRL + layer pruning: (a) heatmap, (b) Pareto per dim- F4. Full pipeline: (a) 3D surface slices, (b) Pareto overlay — KEY FIGURE- F5. Recovery epoch scaling per MRL dim- F6. Group size: (a) g vs PPL per dim, (b) effective bpp- F7. Activation × pruning: (a) grouped bars per dim, (b) degradation ratio- F8. MRL representation analysis: (a) CKA heatmap, (b) gen quality vs dim- F9. Hero figure: full deployment menu from single checkpoint--================================================================================-SECTION 6 — COMPUTE BUDGET-================================================================================--Per-run cost (MAIN-1024, 3 epochs, 1M samples, 4x TPU v5 lite):- ~15 minutes training + ~5 minutes MRL evaluation (5 dims × val + quant)--Estimated total runs:- Group 1: 2 configs × 3 seeds = 6 runs- Group 2: 4 ratios × 3 seeds = 12 runs- Group 3: 4 ratios × 3 seeds = 12 runs- Group 4: 9 pairs × 3 seeds + 4 pairs × 3 seeds (recov)= 39 runs- Group 5: 3 schedules × 3 seeds = 9 runs- Group 6: 5 epoch counts × 3 seeds = 15 runs- Group 7: 4 methods × 2 (QAT on/off) × 3 seeds = 24 runs- Group 8: 4 sizes × 3 configs × 3 seeds = 36 runs- Group 9: 3 act × (3 + 6 + 3) × 3 seeds = 36 runs- Group 10: 3 weighting strategies × 3 seeds = 9 runs- Group 11: evaluation only (reuses checkpoints) = 0 runs- ────────────────────────────────────────────────────────────────────- Total: 198 runs--At ~20 min/run: ~66 TPU-hours (manageable on v5litepod-4)--================================================================================-SECTION 7 — ABLATION CHECKLIST-================================================================================--Before submission, verify these are addressed:-- [ ] Statistical significance: 3 seeds, report mean +/- std- [ ] MRL scaling: smooth degradation curve from full dim to smallest- [ ] MRL + compression composability: all three axes combine gracefully- [ ] Matched comparisons: compare methods at SAME effective param count- [ ] Schedule sensitivity: staged vs reverse vs simultaneous- [ ] Recovery saturation: diminishing returns on fine-tuning epochs- [ ] QAT compatibility: sparsity + quantization + MRL are not antagonistic- [ ] Pruning bias: report which layers (enc vs dec) get pruned- [ ] Generation quality at each MRL dim, not just perplexity- [ ] Group size sensitivity: optimal g, interaction with MRL dim- [ ] Activation × pruning × MRL dim: three-way interaction- [ ] MRL representation analysis: CKA, subspace structure- [ ] Honest limitations: sparse matmuls don't accelerate on current JAX;- MRL dimension reduction requires custom kernel for real speedup- [ ] Comparison to baselines: training K separate small models vs one MRL--================================================================================-SECTION 8 — EXECUTION ORDER-================================================================================--Phase 1 (Day 1): Groups 1 + 2 + 3 (30 runs, ~10 hrs)- Baselines, sparsity-only, layer-prune-only, all with MRL evaluation.- Deliverable: MRL scaling curves and single-axis Pareto frontiers.--Phase 2 (Day 2): Group 4 (39 runs, ~13 hrs)- Full pipeline grid. This produces the key result.- Deliverable: multi-axis Pareto and deployment menu.--Phase 3 (Day 3): Groups 5 + 6 + 7 (48 runs, ~16 hrs)- Schedule, recovery, QAT ablations.- Deliverable: complete ablation tables.--Phase 4 (Day 4): Groups 8 + 9 (72 runs, ~24 hrs)- Group size and activation interaction studies.- Deliverable: granularity and activation robustness analysis.--Phase 5 (Day 5): Groups 10 + 11 + writing (9 runs + eval)- MRL analysis, inference benchmarks, paper draft.- Deliverable: hero figure, all tables populated.--================================================================================-SECTION 9 — PAPER NARRATIVE-================================================================================--Framing: "Train once, deploy anywhere."--A single training run with Matryoshka loss produces a checkpoint that can be-deployed at any dimension slice. Each slice can be further compressed with-group-wise sparsification and layer pruning. The result is a deployment MENU:-- ┌─────────────────────────────────────────────────────────────────────┐- │ Deployment option Est. params Val PPL Quant PPL │- │ ───────────────────────────────────────────────────────────────── │- │ d'=1024, full ~44M X.XX X.XX │- │ d'=512, full ~11M X.XX X.XX │- │ d'=512, s=0.33 ~7M X.XX X.XX │- │ d'=256, full ~3M X.XX X.XX │- │ d'=256, s=0.33 ~2M X.XX X.XX │- │ d'=256, s=0.33, l=0.25 ~1.5M X.XX X.XX │- │ d'=128, full ~0.8M X.XX X.XX │- │ d'=128, s=0.33 ~0.5M X.XX X.XX │- │ d'=64, full ~0.2M X.XX X.XX │- │ d'=64, s=0.33, l=0.25 ~0.1M X.XX X.XX │- └─────────────────────────────────────────────────────────────────────┘--Key claims:- 1. MRL dimension pruning is nearly free — minimal degradation at 2x reduction- 2. Sparsification and layer pruning compose multiplicatively with MRL- 3. A single checkpoint spans 100x+ parameter range- 4. INT4 QAT is compatible with all three compression axes- 5. MemoryMixer encoder is more compressible than standard transformer encoder--================================================================================diff --git a/experiements/tokenizer.txt b/experiements/tokenizer.txtdeleted file mode 100644index e69de29..0000000diff --git a/experiements/training.txt b/experiements/training.txtnew file mode 100644index 0000000..706b729--- /dev/null+++ b/experiements/training.txt@@ -0,0 +1,6 @@+================================================================================+Task: Design the training strategy & loss functions for training a transcribe+tools model.+Deliverable: A well-executed ablation of all strategies against SOTA.+Assignee: Karen Mosoyan & Satyajit Kumar+Deadline: 7th March+================================================================================diff --git a/src/cli.py b/src/cli.pyindex 51803d6..b8b2255 100644--- a/src/cli.py+++ b/src/cli.py@@ -11,7 +11,7 @@ HELP = """│ ││ train ││ --toy Use toy config for quick iteration │- │ --epochs INT Training epochs (default: 2) │+ │ --epochs INT Training epochs (default: 1) ││ --batch-size INT Batch size (default: 32) ││ --lr FLOAT AdamW learning rate (default: 3e-4) ││ --muon-lr FLOAT Muon learning rate (default: 0.02) │@@ -26,6 +26,9 @@ HELP = """│ --sparsity-ratio FLOAT Block prune ratio (default: 0.33) ││ --layer-prune-ratio FL Layer prune ratio (default: 0.0) ││ --group-size INT Quant/prune group size (default: 32) │+ │ --prune-interval INT Steps between mask updates (def: 100) │+ │ --prune-start-frac FL Start pruning at this frac (def: 0.33) │+ │ --prune-end-frac FL Lock mask at this frac (def: 0.67) ││ --activation STR drelu|swiglu|geglu (default: drelu) ││ --warmup-ratio FLOAT LR warmup ratio (default: 0.05) ││ --eval-every INT Val eval interval (default: 1000) │@@ -34,11 +37,6 @@ HELP = """│ --checkpoint-dir DIR Checkpoint directory ││ --seed INT Random seed (default: 42) ││ │- │ sweep │- │ --sweep-config PATH Sweep YAML config │- │ --project STR W&B project name (default: needle-v1) │- │ --count INT Number of trials (default: 20) │- │ ││ run ││ --checkpoint PATH Path to model checkpoint (required) ││ --prompts STR [...] One or more prompts to continue │@@ -85,11 +83,11 @@ TOY_CONFIG = {MAIN_CONFIG = {"num_heads": 4,- "num_layers": 12,+ "num_layers": 24,"num_dec_layers": 4,"max_enc_len": 256,"max_dec_len": 256,- "max_samples": 1000000,+ "max_samples": None,}@@ -113,7 +111,7 @@ def main():p = sub.add_parser("train", add_help=False)p.add_argument("--toy", action="store_true")p.add_argument("--checkpoint", type=str, default=None)- p.add_argument("--epochs", type=int, default=2)+ p.add_argument("--epochs", type=int, default=1)p.add_argument("--batch-size", type=int, default=32)p.add_argument("--lr", type=float, default=3e-4)p.add_argument("--muon-lr", type=float, default=0.02)@@ -131,13 +129,19 @@ def main():p.add_argument("--seed", type=int, default=42)p.add_argument("--eval-every", type=int, default=1000)p.add_argument("--max-eval-samples", type=int, default=None)- p.add_argument("--sparsity-ratio", type=float, default=0.33)+ p.add_argument("--sparsity-ratio", type=float, default=0.5)p.add_argument("--layer-prune-ratio", type=float, default=0.0)p.add_argument("--group-size", type=int, default=32)+ p.add_argument("--prune-interval", type=int, default=100,+ help="Steps between mask updates during gradual pruning (default: 100)")+ p.add_argument("--prune-start-frac", type=float, default=0.33,+ help="Fraction of epoch to train before starting gradual pruning (default: 0.33)")+ p.add_argument("--prune-end-frac", type=float, default=0.67,+ help="Fraction of epoch at which pruning finishes and mask locks (default: 0.67)")p.add_argument("--activation", type=str, default="drelu", choices=["drelu", "swiglu", "geglu"])p.add_argument("--num-memory-slots", type=int, default=64)- p.add_argument("--mrl-dims", type=int, nargs="*", default=[512, 256, 128, 64],- help="MRL dimension pruning targets (default: 512 256 128 64)")+ p.add_argument("--mrl-dims", type=int, nargs="*", default=[1024, 512, 256, 128, 64],+ help="MRL dimension pruning targets (default: 1024 512 256 128 64)")p = sub.add_parser("run", add_help=False)p.add_argument("--checkpoint", type=str, required=True)@@ -156,25 +160,6 @@ def main():p.add_argument("--temperature", type=float, default=0.8)p.add_argument("--throughput-runs", type=int, default=10)- p = sub.add_parser("sweep", add_help=False)- p.add_argument("--sweep-config", type=str, default="sweep.yaml")- p.add_argument("--project", type=str, default="needle-v1")- p.add_argument("--count", type=int, default=20)- p.add_argument("--epochs", type=int, default=1)- p.add_argument("--batch-size", type=int, default=16)- p.add_argument("--lr", type=float, default=3e-4)- p.add_argument("--muon-lr", type=float, default=0.02)- p.add_argument("--d-model", type=int, default=128)- p.add_argument("--num-heads", type=int, default=4)- p.add_argument("--num-layers", type=int, default=2)- p.add_argument("--max-enc-len", type=int, default=128)- p.add_argument("--max-dec-len", type=int, default=128)- p.add_argument("--max-samples", type=int, default=20000)- p.add_argument("--warmup-ratio", type=float, default=0.05)- p.add_argument("--dtype", type=str, default="bfloat16", choices=["float32", "bfloat16"])- p.add_argument("--checkpoint-dir", type=str, default="checkpoints")- p.add_argument("--seed", type=int, default=42)-p = sub.add_parser("evaluate", add_help=False)p.add_argument("--checkpoint", type=str, required=True)p.add_argument("--benchmarks", type=str, nargs="*",@@ -222,9 +207,6 @@ def main():_apply_train_defaults(args)from .train import traintrain(args)- elif args.command == "sweep":- from .train import sweep- sweep(args)elif args.command == "run":from .run import main as run_mainrun_main(args)diff --git a/src/data.py b/src/data.pyindex 7c572b8..9cadde8 100644--- a/src/data.py+++ b/src/data.py@@ -1,4 +1,5 @@import hashlib+import multiprocessing as mpimport osos.environ["TRANSFORMERS_NO_ADVISORY_WARNINGS"] = "1"os.environ["HF_HUB_DISABLE_TELEMETRY"] = "1"@@ -63,6 +64,23 @@ class NeedleTokenizer:return {"input_ids": all_ids}+_worker_sp = None+_worker_max_len = None+++def _init_worker(model_path, max_length):+ """Initializer for multiprocessing pool — loads SP model once per worker."""+ global _worker_sp, _worker_max_len+ _worker_sp = spm.SentencePieceProcessor()+ _worker_sp.Load(model_path)+ _worker_max_len = max_length+++def _tokenize_chunk(texts):+ """Encode a chunk of texts in a worker process."""+ return [_worker_sp.Encode(t, out_type=int)[:_worker_max_len] for t in texts]++def train_tokenizer(vocab_size=8192, max_samples=None):"""Train a SentencePiece BPE tokenizer on TinyStories."""model_path = TOKENIZER_PREFIX + ".model"@@ -151,47 +169,54 @@ def prepare_encoder_decoder_pairs(ds, tokenizer, max_enc_len=128, max_dec_len=12bos_id = tokenizer.eos_token_idmax_total = max_enc_len + max_dec_len- print("Tokenizing...")+ # --- Parallel tokenization ---texts = list(ds["text"])- all_tokens = tokenizer(- texts, truncation=True, max_length=max_total,- )["input_ids"]-- n = len(all_tokens)- enc_inputs = np.full((n, max_enc_len), pad_id, dtype=np.int32)- dec_inputs = np.full((n, max_dec_len), pad_id, dtype=np.int32)- dec_targets = np.full((n, max_dec_len), pad_id, dtype=np.int32)-- kept = 0- for tokens in tqdm(all_tokens, desc="Building pairs"):- if len(tokens) < 6:- continue-- split_point = max(2, int(len(tokens) * split_ratio))- enc_tokens = tokens[:split_point][:max_enc_len]- dec_tokens = tokens[split_point:][:max_dec_len]-- if len(dec_tokens) < 2:- continue-- enc_len = len(enc_tokens)- dec_len = len(dec_tokens)- enc_inputs[kept, :enc_len] = enc_tokens- dec_inputs[kept, 0] = bos_id- dec_inputs[kept, 1:dec_len] = dec_tokens[:dec_len - 1]- dec_targets[kept, :dec_len] = dec_tokens- kept += 1-- enc_inputs = enc_inputs[:kept]- dec_inputs = dec_inputs[:kept]- dec_targets = dec_targets[:kept]+ num_workers = min(os.cpu_count() or 1, 8)+ model_path = TOKENIZER_PREFIX + ".model"+ chunk_size = max(1, len(texts) // (num_workers * 4))+ chunks = [texts[i:i + chunk_size] for i in range(0, len(texts), chunk_size)]++ print(f"Tokenizing ({num_workers} workers)...")+ with mp.Pool(num_workers, initializer=_init_worker,+ initargs=(model_path, max_total)) as pool:+ results = pool.map(_tokenize_chunk, chunks)+ all_tokens = [tok for chunk in results for tok in chunk]++ # --- Pre-filter by length to reduce loop iterations ---+ lengths = np.array([len(t) for t in all_tokens])+ valid_mask = lengths >= 6+ valid_indices = np.where(valid_mask)[0]++ split_points = np.maximum(2, (lengths[valid_indices] * split_ratio).astype(np.int32))+ enc_lens = np.minimum(split_points, max_enc_len)+ dec_lens = np.minimum(lengths[valid_indices] - split_points, max_dec_len)+ keep = dec_lens >= 2++ valid_indices = valid_indices[keep]+ split_points = split_points[keep]+ enc_lens = enc_lens[keep]+ dec_lens = dec_lens[keep]++ n_keep = len(valid_indices)+ enc_inputs = np.full((n_keep, max_enc_len), pad_id, dtype=np.int32)+ dec_inputs = np.full((n_keep, max_dec_len), pad_id, dtype=np.int32)+ dec_targets = np.full((n_keep, max_dec_len), pad_id, dtype=np.int32)++ for i, (idx, sp, el, dl) in enumerate(+ zip(valid_indices, split_points, enc_lens, dec_lens)+ ):+ tokens = all_tokens[idx]+ enc_inputs[i, :el] = tokens[:sp][:max_enc_len]+ dec_inputs[i, 0] = bos_id+ dec_inputs[i, 1:dl] = tokens[sp:][:dl - 1]+ dec_targets[i, :dl] = tokens[sp:][:max_dec_len]# Save cacheos.makedirs(CACHE_DIR, exist_ok=True)np.save(cache_path + "_enc.npy", enc_inputs)np.save(cache_path + "_dec_in.npy", dec_inputs)np.save(cache_path + "_dec_tgt.npy", dec_targets)- print(f"Cached data to {CACHE_DIR}/{cache_id}")+ print(f"Cached {n_keep:,} pairs to {CACHE_DIR}/{cache_id}")return enc_inputs, dec_inputs, dec_targetsdiff --git a/src/train.py b/src/train.pyindex d46b551..7d4112c 100644--- a/src/train.py+++ b/src/train.py@@ -205,6 +205,56 @@ def _block_prune_params(params, prune_ratio=0.25, group_size=32):return jax.tree_util.tree_map_with_path(_prune_weight, params)+def _cubic_sparsity_schedule(step, t_start, t_end, s_final):+ """Cubic sparsity ramp (Zhu & Gupta 2017). Returns target sparsity at *step*."""+ if step < t_start:+ return 0.0+ if step >= t_end:+ return s_final+ frac = (step - t_start) / (t_end - t_start)+ return s_final * (1.0 - (1.0 - frac) ** 3)+++def _make_prune_mask(params, sparsity, group_size):+ """Compute block-prune mask entirely on-device (no numpy round-trips).++ Returns (pruned_params, binary_mask).+ """+ # Pass 1: collect block L1 scores as JAX arrays+ all_scores = []+ for _, leaf in jax.tree_util.tree_leaves_with_path(params):+ if leaf.ndim != 2:+ continue+ in_feat, out_feat = leaf.shape+ gs = min(group_size, in_feat)+ pad = (gs - in_feat % gs) % gs+ w = jnp.pad(leaf, ((0, pad), (0, 0))) if pad else leaf+ scores = jnp.sum(jnp.abs(w.reshape(-1, gs, out_feat)), axis=1).ravel()+ all_scores.append(scores)++ if not all_scores:+ ones = jax.tree.map(jnp.ones_like, params)+ return params, ones++ threshold = jnp.percentile(jnp.concatenate(all_scores), sparsity * 100)++ # Pass 2: build per-leaf binary mask+ def _leaf_mask(path, leaf):+ if leaf.ndim != 2:+ return jnp.ones_like(leaf)+ in_feat, out_feat = leaf.shape+ gs = min(group_size, in_feat)+ pad = (gs - in_feat % gs) % gs+ w = jnp.pad(leaf, ((0, pad), (0, 0))) if pad else leaf+ w_grouped = w.reshape(-1, gs, out_feat)+ block_keep = (jnp.sum(jnp.abs(w_grouped), axis=1, keepdims=True) > threshold)+ return jnp.broadcast_to(block_keep, w_grouped.shape).reshape(-1, out_feat)[:in_feat].astype(jnp.float32)++ mask = jax.tree_util.tree_map_with_path(_leaf_mask, params)+ pruned = jax.tree.map(lambda w, m: w * m, params, mask)+ return pruned, mask++def _layer_prune(params, config, layer_prune_ratio=0.25):"""Remove the lowest-scoring encoder/decoder blocks by L1 magnitude.@@ -529,37 +579,22 @@ def train(args):sparsity_ratio = getattr(args, "sparsity_ratio", 0.0)layer_prune_ratio = getattr(args, "layer_prune_ratio", 0.0)prune_mask = None+ gradual_pruning_done = False- weight_prune_epoch = 1 if sparsity_ratio > 0 else -1- layer_prune_epoch = (2 if sparsity_ratio > 0 else 1) if layer_prune_ratio > 0 else -1-- for epoch in range(args.epochs):- if epoch == weight_prune_epoch and prune_mask is None:- print(f"\nGroup-wise sparsification: {sparsity_ratio*100:.0f}% of blocks (group_size={_GROUP_SIZE})...")- pruned_params = _block_prune_params(jax_utils.unreplicate(ema_params), prune_ratio=sparsity_ratio, group_size=_GROUP_SIZE)+ prune_interval = getattr(args, "prune_interval", 100)+ prune_start_frac = getattr(args, "prune_start_frac", 0.33)+ prune_end_frac = getattr(args, "prune_end_frac", 0.67)- pruned_np = jax.tree.map(np.array, pruned_params)- total_p = sum(x.size for x in jax.tree.leaves(pruned_np))- zero_p = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(pruned_np))- print(f" Post-prune sparsity: {zero_p/total_p*100:.2f}% ({zero_p:,}/{total_p:,} near-zero)")+ weight_prune_epoch = 0 if sparsity_ratio > 0 else -1+ layer_prune_epoch = (1 if sparsity_ratio > 0 else 0) if layer_prune_ratio > 0 else -1- val_causal = make_causal_mask(args.max_dec_len)- total_loss, total_toks = 0.0, 0.0- for vb in get_batches(val_enc, val_dec_in, val_dec_tgt, args.batch_size, shuffle=False):- vl, vt = val_loss_fn(pruned_params, vb[0], vb[1], vb[2], val_causal)- total_loss += float(vl)- total_toks += float(vt)- prune_val_ppl = float(math.exp(total_loss / max(total_toks, 1)))- print(f" Post-prune val ppl: {prune_val_ppl:.2f} (was {last_val_ppl:.2f})" if last_val_ppl is not None else f" Post-prune val ppl: {prune_val_ppl:.2f}")-- state = jax_utils.unreplicate(state)- state = state.replace(params=pruned_params)- ema_params = jax.tree.map(jnp.copy, pruned_params)- prune_mask = jax.tree.map(lambda w: (jnp.abs(w) > 1e-8).astype(jnp.float32), pruned_params)- prune_mask = jax_utils.replicate(prune_mask)- state = jax_utils.replicate(state)- ema_params = jax_utils.replicate(ema_params)- print(f"\nContinuing training with sparsity mask locked...\n")+ for epoch in range(args.epochs):+ if epoch == weight_prune_epoch and not gradual_pruning_done:+ t_start = int(num_batches * prune_start_frac)+ t_end = int(num_batches * prune_end_frac)+ print(f"\nGradual magnitude pruning: 0% -> {sparsity_ratio*100:.0f}% over epoch {epoch+1} "+ f"(steps {t_start}-{t_end}/{num_batches}, interval={prune_interval}, group_size={_GROUP_SIZE})")+ epoch_step = 0if epoch == layer_prune_epoch:print(f"\nLayer pruning: removing {layer_prune_ratio*100:.0f}% of layers...")@@ -609,6 +644,15 @@ def train(args):causal_mask,)+ # --- Gradual magnitude pruning (epoch 1 only) ---+ if epoch == weight_prune_epoch and not gradual_pruning_done:+ epoch_step += 1+ current_sparsity = _cubic_sparsity_schedule(epoch_step, t_start, t_end, sparsity_ratio)+ if epoch_step >= t_start and epoch_step % prune_interval == 0 and current_sparsity > 0:+ ema_unr = jax_utils.unreplicate(ema_params)+ _, mask = _make_prune_mask(ema_unr, current_sparsity, _GROUP_SIZE)+ prune_mask = jax_utils.replicate(mask)+if prune_mask is not None:state = state.replace(params=jax.tree.map(lambda w, m: w * m, state.params, prune_mask))ema_params = jax.tree.map(lambda w, m: w * m, ema_params, prune_mask)@@ -628,7 +672,12 @@ def train(args):total_toks += float(vt)del _eval_paramslast_val_ppl = float(math.exp(total_loss / max(total_toks, 1)))- pbar.set_postfix(loss=f"{loss_val:.4f}", val_ppl=f"{last_val_ppl:.2f}" if last_val_ppl is not None else "?")++ # --- tqdm postfix with sparsity during epoch 1 ---+ postfix = {"loss": f"{loss_val:.4f}", "val_ppl": f"{last_val_ppl:.2f}" if last_val_ppl is not None else "?"}+ if epoch == weight_prune_epoch and not gradual_pruning_done:+ postfix["sparsity"] = f"{current_sparsity*100:.1f}%"+ pbar.set_postfix(**postfix)if use_wandb:log_dict = {@@ -639,10 +688,29 @@ def train(args):"train/tokens_per_sec": tokens_per_batch / dt,"train/step": global_step,}+ if epoch == weight_prune_epoch and not gradual_pruning_done:+ log_dict["train/scheduled_sparsity"] = current_sparsityif global_step % eval_every == 0 or global_step == total_steps:log_dict["val/ppl"] = last_val_pplwandb.log(log_dict)+ # --- Lock mask after epoch 1 gradual pruning ---+ if epoch == weight_prune_epoch and not gradual_pruning_done:+ gradual_pruning_done = True+ if prune_mask is None:+ # Edge case: epoch had fewer steps than t_start+ ema_unr = jax_utils.unreplicate(ema_params)+ _, mask = _make_prune_mask(ema_unr, sparsity_ratio, _GROUP_SIZE)+ prune_mask = jax_utils.replicate(mask)+ state = state.replace(params=jax.tree.map(lambda w, m: w * m, state.params, prune_mask))+ ema_params = jax.tree.map(lambda w, m: w * m, ema_params, prune_mask)+ final_pruned = jax.tree.map(np.array, jax_utils.unreplicate(ema_params))+ total_p = sum(x.size for x in jax.tree.leaves(final_pruned))+ zero_p = sum(int(np.sum(np.abs(x) < 1e-6)) for x in jax.tree.leaves(final_pruned))+ print(f"\n Gradual pruning complete — mask locked.")+ print(f" Final sparsity: {zero_p/total_p*100:.2f}% ({zero_p:,}/{total_p:,} near-zero)")+ del final_pruned+epoch_avg_loss = sum(losses) / len(losses) if losses else float("nan")final_loss = losses[-1] if losses else float("nan")final_ppl = math.exp(final_loss) if not math.isnan(final_loss) else float("nan")@@ -753,44 +821,3 @@ def train(args):print("\nTraining complete.")-def sweep(args):- import wandb- import yaml-- with open(args.sweep_config) as f:- sweep_config = yaml.safe_load(f)-- def sweep_train():- wandb.init(project=args.project)- for key, value in dict(wandb.config).items():- if hasattr(args, key):- setattr(args, key, value)- args.wandb = True- train(args)- wandb.finish()-- sweep_id = wandb.sweep(sweep_config, project=args.project)- wandb.agent(sweep_id, function=sweep_train, count=args.count)---def parse_args():- parser = argparse.ArgumentParser(description="Train encoder-decoder transformer on TinyStories")- parser.add_argument("--epochs", type=int, default=2)- parser.add_argument("--batch-size", type=int, default=32)- parser.add_argument("--lr", type=float, default=3e-4)- parser.add_argument("--muon-lr", type=float, default=0.02)- parser.add_argument("--d-model", type=int, default=128)- parser.add_argument("--num-heads", type=int, default=4)- parser.add_argument("--num-layers", type=int, default=2)- parser.add_argument("--max-enc-len", type=int, default=128)- parser.add_argument("--max-dec-len", type=int, default=128)- parser.add_argument("--max-samples", type=int, default=10000)- parser.add_argument("--log-every", type=int, default=50)- parser.add_argument("--warmup-ratio", type=float, default=0.05)- parser.add_argument("--checkpoint-dir", type=str, default="checkpoints")- parser.add_argument("--seed", type=int, default=42)- return parser.parse_args()---if __name__ == "__main__":- train(parse_args())diff --git a/sweep.yaml b/sweep.yamldeleted file mode 100644index 6bd0806..0000000--- a/sweep.yaml+++ /dev/null@@ -1,27 +0,0 @@-method: bayes-metric:- name: epoch/avg_loss- goal: minimize-parameters:- lr:- distribution: log_uniform_values- min: 1.0e-5- max: 1.0e-2- muon_lr:- distribution: log_uniform_values- min: 5.0e-3- max: 0.1- warmup_ratio:- distribution: uniform- min: 0.01- max: 0.15- dropout:- values: [0.0, 0.05, 0.1, 0.15, 0.2]- batch_size:- values: [16, 32, 64]- d_model:- values: [64, 128, 256]- num_layers:- values: [2, 3, 4]- num_heads:- values: [2, 4, 8]