Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

add notes

HenryNdubuakuee916036a474
diff --git a/notes/architecture.txt b/notes/architecture.txt
new file mode 100644
index 0000000..c421403
--- /dev/null
+++ b/notes/architecture.txt
@@ -0,0 +1,720 @@
+Co-Designed Architecture & Training Recipe for Small-Scale Tool Calling
+=======================================================================
+
+## 1. Thesis
+
+No single component is novel in isolation. The contribution is a
+co-designed system: an encoder-decoder architecture trained with a
+multi-objective recipe (generation + contrastive + Matryoshka +
+structured weighting + QAT + pruning) that produces a small model
+competitive with much larger decoder-only models on tool calling.
+
+The architecture choices are interdependent:
+ - Encoder-decoder enables shared-encoder retrieval (see tool-selection.txt)
+ - Matryoshka FFN enables elastic deployment at multiple compute budgets
+ - Structured loss weighting targets the actual error distribution
+ - QAT + pruning enable deployment on edge devices
+ - Constrained decoding exploits known schemas (see tries.txt)
+
+Each piece alone is known. The system is the contribution.
+
+
+## 2. Architecture
+
+### 2a. Why Encoder-Decoder (Not Decoder-Only)
+
+Almost all tool-calling models are decoder-only. We use encoder-decoder
+because tool calling has a natural asymmetry:
+
+ Input (encode once): Output (generate):
+ ┌──────────────────────┐ ┌──────────────────────┐
+ │ Query: "What's the │ │[{"name":"get_weather"│
+ │ weather in SF?" │ │ "arguments": │
+ │ <tools> │ │ {"location":"SF"}} │
+ │ [{"name":"get_weather│ │ ] │
+ │ "parameters":{...}}│ └──────────────────────┘
+ │ {"name":"get_stock" │
+ │ "parameters":{...}}│
+ │ ] │
+ └──────────────────────┘
+
+ Encoder input = context (query + tool definitions) — processed once
+ Decoder output = structured generation (JSON tool calls) — autoregressive
+
+Benefits:
+ (a) The encoder processes all tools in parallel via bidirectional
+ self-attention. A decoder-only model processes tools left-to-right,
+ which means later tools attend to earlier tools but not vice versa.
+ For discriminating between semantically similar tools (search_flights
+ vs search_hotels), bidirectional context helps.
+
+ (b) Encoder output is reused across all decoder steps via cross-attention.
+ In a decoder-only model, KV cache stores the full context, but the
+ representation is computed left-to-right with causal masking — no
+ opportunity for the context to self-refine bidirectionally.
+
+ (c) The encoder can serve double duty: cross-attention for generation
+ AND contrastive retrieval for tool selection (see tool-selection.txt).
+ A decoder-only model would need a separate retrieval module.
+
+ (d) Smaller parameter budget. With the same d_model and layer count,
+ an encoder-decoder can encode context more efficiently because
+ the encoder layers don't need causal masking, cross-attention
+ projections, or autoregressive logic.
+
+
+### 2b. Zero-Centered RMSNorm (ZCRMSNorm)
+
+Standard RMSNorm: γ * x / RMS(x), with γ initialized to 1.
+Our ZCRMSNorm: (1 + γ) * x / RMS(x), with γ initialized to 0.
+
+ class ZCRMSNorm(nn.Module):
+ def __call__(self, x):
+ scale = self.param("scale", zeros, (x.shape[-1],))
+ rms = sqrt(mean(x^2, axis=-1, keepdims=True) + eps)
+ return (1 + scale) * x / rms
+
+At initialization, (1 + 0) * x / RMS(x) = x / RMS(x) — the layer
+normalizes but applies no learned scaling. This is similar in spirit
+to ReZero (Bachlechner et al. 2020) and fixup initialization: at
+init, each sub-layer's contribution is minimal, and the residual
+stream dominates. The network starts as a near-identity function
+and gradually learns to use each layer.
+
+Why this matters for small models:
+ - Small models are more sensitive to initialization. Standard RMSNorm
+ with γ=1 can cause early training instability because pre-norm
+ layers immediately rescale activations.
+ - Zero-init means gradients flow cleanly through the residual stream
+ at the start of training, avoiding the "early chaos" period.
+
+Applied everywhere: encoder pre-norm, decoder pre-norm, QK-norm,
+ConvModule norm. Consistent zero-centering throughout.
+
+
+### 2c. Grouped Query Attention (GQA) with QK-Norm
+
+Standard multi-head attention with grouped KV heads:
+
+ Q: (B, num_heads, T, head_dim) — full heads
+ K: (B, num_kv_heads, T, head_dim) — grouped (repeated for attention)
+ V: (B, num_kv_heads, T, head_dim) — grouped
+
+ num_heads=8, num_kv_heads=2 → 4:1 ratio, 75% KV parameter savings
+
+QK-norm: apply ZCRMSNorm to Q and K after projection, before attention.
+
+ q = ZCRMSNorm(q_proj(x))
+ k = ZCRMSNorm(k_proj(x))
+ attn = softmax(q @ k^T / sqrt(d_head))
+
+QK-norm prevents attention logit growth as training progresses (the
+"logit drift" problem observed in large-scale training). With zero-init
+normalization, Q and K start with unit-scale dot products and gradually
+learn scale — stable from step 1.
+
+This combination (GQA + QK-norm + ZCRMSNorm) is individually known
+(Gemma 2, ViT-22B use QK-norm; GQA from Ainslie et al. 2023) but the
+zero-centered variant on Q/K is our specific choice.
+
+
+### 2d. DReLU Activation
+
+Three activation options: SwiGLU, GeGLU, DReLU. Default is DReLU.
+
+ SwiGLU: silu(gate) * up — standard, smooth
+ GeGLU: gelu(gate) * up — smoother, slightly more expensive
+ DReLU: relu(gate) * relu(up) — simplest, sparsest
+
+ h = relu(W_gate @ x) * relu(W_up @ x)
+ out = W_down @ h
+
+DReLU applies ReLU to both gate and up projections independently.
+This produces the sparsest activations: any input where either gate
+OR up projection is negative gets zeroed. Empirically, ~60-70% of
+hidden units are zero at any given step.
+
+Why DReLU for tool calling:
+ - Activation sparsity enables hardware-level speedups (skip zero
+ multiplications). Critical for edge deployment.
+ - Works synergistically with magnitude pruning: sparse activations
+ mean many weights have low magnitude → cleaner pruning signal.
+ - Cheaper than SwiGLU/GeGLU: no sigmoid/erf computation.
+ - For structured output (JSON), the model doesn't need smooth
+ gradients for nuanced language — it needs sharp, discriminative
+ features for selecting the right tool name and argument keys.
+
+
+### 2e. Matryoshka FFN
+
+Train a single model that contains multiple sub-models at different
+FFN widths. At inference, choose the width based on compute budget.
+
+ Full model: d_ff = 2048 (all neurons active)
+ 2x sub-model: d_ff = 1024 (first 1024 neurons)
+ 4x sub-model: d_ff = 512 (first 512 neurons)
+
+Implementation: prefix masking on FFN hidden dimension.
+
+ def __call__(self, x, ffn_mask=None):
+ gate = gate_proj(x) # (B, T, d_ff)
+ up = up_proj(x) # (B, T, d_ff)
+ h = relu(gate) * relu(up)
+ if ffn_mask is not None:
+ h = h * ffn_mask # zero out neurons beyond width
+ return down_proj(h)
+
+During training, each batch is split across widths:
+
+ Batch (B=32):
+ items 0-10: full width (d_ff=2048) — all neurons
+ items 11-21: 2x width (d_ff=1024) — first half
+ items 22-31: 4x width (d_ff=512) — first quarter
+
+ All items share the same forward pass (single jit-compiled function).
+ The ffn_mask varies per item. No separate models, no distillation.
+
+The prefix constraint is key: sub-models always use the FIRST k neurons.
+This means the first 512 neurons must be useful on their own, the first
+1024 must be useful, etc. The training objective forces the model to
+organize information by importance along the FFN dimension — most
+important features in early neurons.
+
+At inference:
+ - Edge device: use 4x sub-model (512 neurons, ~25% FFN compute)
+ - Server: use full model (2048 neurons, full quality)
+ - Same checkpoint, same parameters, different masks
+
+Why not distillation:
+ - Distillation requires training N separate models
+ - Matryoshka trains one model, gets N sub-models
+ - Sub-models share attention weights — only FFN width changes
+ - Parameter estimation for any sub-model is straightforward
+
+
+### 2f. Shared Encoder with Contrastive Head
+
+See tool-selection.txt for full details. Summary:
+
+ Encoder → mean pool → Dense(d_model, 128) → L2 normalize → cosine similarity
+
+ Same encoder serves:
+ (a) Cross-attention in decoder (generation)
+ (b) Contrastive retrieval (tool selection from large catalogs)
+
+ Adds <0.01% parameters (one 512x128 matrix + learnable temperature).
+
+
+### 2g. Conformer Convolution Module (Optional)
+
+When conv_kernel_size > 0, each encoder block adds a convolution module
+between self-attention and FFN:
+
+ self-attn → ConvModule → FFN
+
+ ConvModule:
+ pointwise up (d → 2d) → GLU gate → depthwise conv (k=31) →
+ ZCRMSNorm → pointwise down (d → d)
+
+This is the Conformer (Gulati et al. 2020) pattern. The depthwise
+convolution captures local patterns (token n-grams in tool definitions,
+local acoustic patterns in speech). Complementary to global self-attention.
+
+Primary motivation: speech pathway benefits significantly from local
+convolution. For text-only, conv_kernel_size=0 (disabled).
+
+
+### 2h. Speech Pathway
+
+Optional speech input via FastConformer-style 8x subsampling:
+
+ Input: (B, T, 80) mel spectrogram
+ Output: (B, T//8, d_model) features fed to shared encoder
+
+ Subsampling (3 stages of stride-2 = 8x total):
+ Conv2D(1→C, k=3, s=2) + ReLU
+ DW-Conv(C, k=3, s=2) + PW-Conv(C→C) + ReLU
+ DW-Conv(C, k=3, s=2) + PW-Conv(C→C) + ReLU
+ Reshape + Linear → d_model
+
+ SpecAugment (training only):
+ 2 time masks (max 100 frames)
+ 1 frequency mask (max 27 bins)
+
+The speech subsystem converts audio to the same representation space
+as text. After subsampling, the encoder processes speech features
+identically to text embeddings — same attention, same FFN, same
+contrastive head. Unified encoder for both modalities.
+
+
+### 2i. Memory-Efficient Layer Stacking
+
+Both encoder and decoder use nn.scan with remat:
+
+ ScanBlock = nn.scan(
+ nn.remat(BlockBody),
+ variable_axes={"params": 0}, # stack params along axis 0
+ split_rngs={"params": True},
+ length=num_layers,
+ )
+
+ - nn.scan: shares the block's Python code across layers, stacking
+ parameters as (num_layers, ...) tensors. Reduces compilation time.
+ - nn.remat: recomputes activations during backward pass instead of
+ storing them. Trades ~30% more compute for ~60% less memory.
+
+For a small model this matters less, but it enables scaling experiments
+with deeper models on the same hardware.
+
+
+## 3. Training Recipe
+
+### 3a. Muon + Adam Split Optimizer
+
+Not all parameters benefit from the same optimizer. We split:
+
+ Muon optimizer: all Dense kernels (2D+ tensors)
+ - Orthogonalize gradients via Newton-Schulz polar decomposition
+ - Nesterov momentum (μ=0.95)
+ - Weight decay 0.01
+ - Separate WSD learning rate schedule
+
+ Adam optimizer: everything else (embeddings, norms, biases, 1D params)
+ - AdamW with β2=0.95 (lower than default 0.999 for stability)
+ - No weight decay
+ - Separate WSD learning rate schedule
+
+Muon's gradient orthogonalization:
+
+ def _newton_schulz(G, steps=5):
+ a, b, c = 3.4445, -4.7750, 2.0315
+ X = G / norm(G)
+ for _ in range(steps):
+ A = X @ X^T
+ B = b*A + c*A@A
+ X = a*X + B@X
+ return X
+
+ This approximates the polar decomposition G = U*S*V^T and returns U*V^T.
+ The gradient is projected onto the nearest orthogonal matrix.
+
+Why Muon for attention/FFN kernels:
+ - Orthogonalized updates prevent weight matrix collapse (rank
+ deficiency) in small models where d_model is limited.
+ - Newton-Schulz is cheap: 5 iterations of matrix multiply, no SVD.
+ - Empirically, Muon converges faster than Adam on projection matrices.
+
+Why Adam for the rest:
+ - Embeddings are lookup tables — orthogonalization doesn't apply.
+ - Norm scales are 1D — orthogonalization is meaningless.
+ - Adam's adaptive learning rates are better for heterogeneous params.
+
+
+### 3b. Warmup-Stable-Decay (WSD) Schedule
+
+ ┌─────────────────────────────────────────────────────────┐
+ │ LR │
+ │ ▲ │
+ │ │ ┌──────────────────────────────┐ │
+ │ │ /│ stable │\ │
+ │ │ / │ │ \ cosine │
+ │ │ / │ │ \ decay │
+ │ │ / │ │ \ │
+ │ │ / │ │ \____ │
+ │ │/ warmup │ │
+ │ └──────┴──────────────────────────────┴────────→ step │
+ │ 0% ~5% 85% 100% │
+ └─────────────────────────────────────────────────────────┘
+
+ - Warmup: linear ramp from 0 to peak (first ~5% of steps)
+ - Stable: hold at peak (majority of training)
+ - Decay: cosine anneal to 5% of peak (last 15% of steps)
+
+Applied separately to Adam and Muon with different peak values.
+WSD is simpler than cosine-with-restarts and matches or beats it
+for fixed-budget training (MiniCPM, Zhai et al. 2022).
+
+
+### 3c. Quantization-Aware Training (QAT)
+
+Every 100 steps, apply fake INT4 quantization to all Dense kernels:
+
+ def _fake_quantize_int4(w, group_size=32):
+ # Group along input dimension
+ w_grouped = w.reshape(num_groups, group_size, out_feat)
+ # Per-group symmetric quantization
+ scale = max(abs(w_grouped), axis=1) / 7.0
+ w_q = clip(round(w_grouped / scale), -8, 7) * scale
+ # Straight-through estimator
+ return w + stop_gradient(w_q - w)
+
+ - Symmetric INT4: 4-bit integers [-8, 7] with per-group scale
+ - Group size 32: each 32 consecutive input features share a scale
+ - STE: forward uses quantized weights, backward uses real gradients
+ - Applied conditionally: only every 100 steps (reduces overhead)
+
+The model learns to be robust to INT4 quantization during training.
+At deployment, weights are actually quantized to INT4 → 4x memory
+reduction, hardware-accelerated INT4 matmul on supporting devices.
+
+Why every 100 steps instead of every step:
+ - QAT adds ~15% overhead per step (extra quantize + forward pass)
+ - Every 100 steps: <0.15% amortized overhead
+ - Weights change slowly — quantization noise from step N is still
+ representative at step N+99
+ - Same rationale as contrastive loss frequency (see tool-selection.txt)
+
+
+### 3d. Group-Aligned Gradual Pruning
+
+See sparsification.txt for full details. Summary:
+
+ - Group-wise magnitude pruning with group_size=32 — same groups as QAT
+ - Cubic gradual schedule: 0% → target over middle third of epoch
+ - Global threshold across all layers and matrices
+ - Mask applied after gradient update, locked after pruning epoch
+
+ The critical co-design: pruned groups are exactly the groups the INT4
+ quantization kernel operates on. A pruned group = zero weights + zero
+ scale factor = zero memory loads + zero MACs. On edge INT4 kernels,
+ 50% group sparsity → 50% fewer memory accesses and 50% fewer compute
+ operations, directly translating to battery savings.
+
+
+### 3e. Structured Per-Token Loss Weighting
+
+Not all tokens in the decoder output are equally important:
+
+ [{"name":"get_weather","arguments":{"location":"San Francisco"}}]
+
+ Token role Weight Rationale
+ ────────── ────── ─────────
+ name value 3.0 Wrong name = completely wrong call
+ arg value 2.0 Wrong value = wrong behavior
+ arg key 1.5 Wrong key = hallucinated parameter
+ structure 1.0 JSON syntax (already learned well)
+
+Implemented as per-token mask in loss computation:
+
+ loss = sum(CE(logits, targets) * token_weights) / sum(token_weights)
+
+The weighting is computed during data preparation by parsing the
+target JSON and labeling each token's role. This focuses gradient
+signal on the tokens that matter most for tool-call correctness.
+
+Why not uniform weighting:
+ - 60-70% of decoder tokens are JSON structure ({, }, [, ], :, ", ,)
+ - The model learns structure quickly (99%+ parse rate)
+ - Under uniform weighting, gradient signal is dominated by structure
+ tokens — the model keeps "improving" on tokens it already gets right
+ - 3x weight on names means 3x gradient signal for the critical tokens
+
+
+### 3f. Z-Loss Regularization
+
+Small auxiliary loss on logit magnitude:
+
+ z_loss = 1e-4 * mean(logsumexp(logits, axis=-1)^2)
+
+Prevents logits from growing unboundedly during training. Without this,
+logits can reach 100+ in magnitude, causing numerical issues in softmax
+and making the model overconfident. Z-loss keeps logits in a reasonable
+range (~10-20) without affecting the argmax (relative ordering preserved).
+
+
+### 3g. Multi-Objective Training Schedule
+
+All objectives in a single training loop, but at different frequencies:
+
+ Every step:
+ - Text generation loss (CE + z-loss)
+ - Matryoshka FFN masking (batch split across widths)
+ - Gradient clip (global norm 1.0)
+ - Muon/Adam update
+
+ Every 100 steps:
+ - QAT (fake INT4 quantization)
+ - Contrastive loss (CLIP-style, weight 0.1)
+
+ During pruning epoch (steps 33%-67%):
+ - Every prune_interval steps: recompute prune mask
+ - Apply mask after each gradient update
+
+ End of pruning epoch:
+ - Lock mask, report final sparsity
+
+This schedule minimizes overhead: the expensive objectives (QAT,
+contrastive) run infrequently, and their signals are low-noise
+enough that per-step computation is unnecessary.
+
+
+## 4. Experiment Design
+
+### Exp 1: Encoder-Decoder vs Decoder-Only [MUST-HAVE]
+
+Setup:
+ - Train encoder-decoder (our architecture) and decoder-only
+ (same total params, same d_model, combined layer count) on
+ identical data with identical recipe
+ - Decoder-only: concatenate [query | tools | answer] as single
+ sequence, causal attention throughout
+ - Match total FLOPs as closely as possible
+
+Metrics:
+ | Architecture | Params | name_f1 | call_f1 | exact_match | value_acc |
+
+ Also report:
+ - Inference latency (tok/s) for both
+ - Encoder reuse efficiency (for enc-dec: encode once, decode many)
+
+Hypothesis:
+ - Enc-dec should win on name_f1 (bidirectional tool encoding)
+ - Enc-dec should win on multi-tool examples (better tool discrimination)
+ - Dec-only may be competitive on single-tool examples
+ - Enc-dec inference faster when generating multiple calls per context
+ (encoder output cached, decoder runs multiple times)
+
+What this shows:
+ - The architectural choice is justified for tool calling
+ - Bidirectional encoding of tool definitions helps discrimination
+
+
+### Exp 2: Full Recipe Ablation [MUST-HAVE]
+
+Setup:
+ - Full recipe: all components enabled
+ - Drop-one ablations (each removes exactly one component):
+ (a) No Muon (Adam-only for all params)
+ (b) No QAT (no fake quantization)
+ (c) No pruning (no magnitude pruning)
+ (d) No structured loss weighting (uniform weights)
+ (e) No Matryoshka (full FFN width only)
+ (f) No contrastive (already in tool-selection.txt Exp 2)
+ (g) No z-loss
+ (h) No DReLU (use SwiGLU instead)
+
+Metrics:
+ | Condition | val_ppl | name_f1 | call_f1 | quant_ppl | sparse_ppl |
+ |---------------------|---------|---------|---------|-----------|------------|
+ | Full recipe | | | | | |
+ | - Muon | | | | | |
+ | - QAT | | | | N/A | |
+ | - Pruning | | | | | N/A |
+ | - Struct. weights | | | | | |
+ | - Matryoshka | | | | | |
+ | - z-loss | | | | | |
+ | - DReLU (+SwiGLU) | | | | | |
+
+Hypothesis:
+ - Full recipe is best on aggregate
+ - Dropping structured weighting hurts name_f1 most (name tokens
+ lose 3x gradient signal)
+ - Dropping Muon hurts convergence speed (need more epochs)
+ - Dropping QAT: val_ppl similar, but quant_ppl much worse
+ - Dropping pruning: val_ppl similar, but sparse_ppl worse
+ - Each component contributes — no single one is dominant
+
+What this shows:
+ - The recipe is co-designed: each piece has a role
+ - No single component carries the result — it's the combination
+
+
+### Exp 3: Matryoshka Accuracy-Efficiency Pareto [MUST-HAVE]
+
+Setup:
+ - Single trained model with Matryoshka FFN
+ - Evaluate at FFN widths: full (2048), 2x (1024), 4x (512), 8x (256)
+ - Compare with separately trained models at each width
+ - Compare with knowledge distillation (full → small)
+
+Metrics:
+ | Method | d_ff | Params | FLOPs | val_ppl | call_f1 |
+ |-----------------|------|--------|-------|---------|---------|
+ | Matryoshka full | 2048 | | | | |
+ | Matryoshka 2x | 1024 | | | | |
+ | Matryoshka 4x | 512 | | | | |
+ | Matryoshka 8x | 256 | | | | |
+ | Separate full | 2048 | | | | |
+ | Separate 2x | 1024 | | | | |
+ | Separate 4x | 512 | | | | |
+ | Separate 8x | 256 | | | | |
+ | Distilled 2x | 1024 | | | | |
+ | Distilled 4x | 512 | | | | |
+
+ Plot: Pareto curve (FLOPs vs call_f1) for each method
+
+Hypothesis:
+ - Matryoshka sub-models are within 1-3% of separately trained models
+ - Matryoshka beats distillation (no teacher-student mismatch)
+ - Single checkpoint, multiple operating points — simpler deployment
+
+What this shows:
+ - Matryoshka is a practical elastic inference solution
+ - The accuracy-efficiency tradeoff is smooth (no cliff)
+ - One model replaces N separately trained models
+
+
+### Exp 4: Optimizer Comparison [STRONG]
+
+Setup:
+ - Muon+Adam (our split): Muon for kernels, Adam for rest
+ - Adam-only: AdamW with same schedule for all params
+ - SGD+momentum: baseline optimizer
+ - Same architecture, same data, same total steps
+
+Metrics:
+ - Training loss curves (convergence speed)
+ - Final val_ppl and call_f1
+ - Wall-clock time per step
+
+Hypothesis:
+ - Muon+Adam converges fastest (fewer steps to same loss)
+ - Adam-only reaches similar final quality but slower
+ - SGD is significantly worse (no adaptive rates for heterogeneous params)
+ - Muon adds negligible wall-clock overhead (Newton-Schulz is 5 matmuls)
+
+What this shows:
+ - Muon accelerates convergence for attention/FFN matrices
+ - The split optimizer is justified: different params need different treatment
+
+
+### Exp 5: QAT + Pruning Compression Analysis [STRONG]
+
+Setup:
+ - Evaluate combinations:
+ Full precision, no pruning (baseline)
+ INT4 quantized (post-training, no QAT)
+ INT4 quantized (with QAT)
+ 50% pruned (no QAT)
+ INT4 + 50% pruned (both, with QAT)
+ - Report model size, inference speed, and accuracy
+
+Metrics:
+ | Condition | Size(MB) | tok/s | val_ppl | call_f1 |
+ |----------------------|----------|-------|---------|---------|
+ | FP16 baseline | | | | |
+ | INT4 (no QAT) | | | | |
+ | INT4 (with QAT) | | | | |
+ | 50% pruned | | | | |
+ | INT4 + 50% pruned | | | | |
+
+ Also: group_size sweep for QAT {16, 32, 64, 128}
+
+Hypothesis:
+ - QAT recovers most of the INT4 quantization loss (quant_ppl gap < 0.5)
+ - Without QAT, INT4 has significant degradation at small model sizes
+ - Pruning + QAT compound: 8x size reduction with <5% accuracy loss
+ - Group size 32 is sweet spot (matches hardware vector width)
+
+What this shows:
+ - QAT and pruning are essential for edge deployment
+ - Co-designed training (both at group_size=32) enables extreme compression
+ - Small models benefit more from QAT than large models (fewer redundant params)
+
+
+### Exp 6: Activation Comparison (DReLU vs SwiGLU vs GeGLU) [STRONG]
+
+Setup:
+ - Three activations, everything else identical
+ - Report: accuracy, activation sparsity, inference speed
+
+Metrics:
+ | Activation | val_ppl | call_f1 | sparsity% | tok/s | quant_ppl |
+
+Hypothesis:
+ - DReLU has highest activation sparsity (~65%)
+ - SwiGLU has best raw val_ppl (smoothest gradients)
+ - DReLU wins on call_f1 (sharper features for structured output)
+ - DReLU wins on tok/s (no sigmoid/erf, sparse activations)
+ - DReLU has best quant_ppl (sparse activations → better quantization)
+
+What this shows:
+ - DReLU is the right choice for tool calling + edge deployment
+ - The sparsity-accuracy tradeoff favors DReLU in this domain
+
+
+### Exp 7: Structured Loss Weight Sensitivity [STRONG]
+
+Setup:
+ - Vary name weight: {1.0, 2.0, 3.0, 5.0, 10.0}
+ - Vary value weight: {1.0, 2.0, 3.0, 5.0}
+ - Key weight fixed at 1.5x
+
+Metrics (heatmap: name_weight x value_weight):
+ - name_f1
+ - call_f1
+ - value_acc
+
+Hypothesis:
+ - name_weight=1 (uniform): worst name_f1
+ - name_weight=3: sweet spot (sufficient signal without overwhelming)
+ - name_weight=10: name_f1 saturates, value_acc degrades (gradient
+ starved on value tokens)
+ - value_weight=2: good balance
+
+What this shows:
+ - Structured weighting matters for structured output
+ - The weights should reflect the error impact hierarchy:
+ wrong name (catastrophic) > wrong value (incorrect) > wrong key (hallucinated)
+
+
+### Exp 8: Training Curriculum Effect [NICE-TO-HAVE]
+
+Setup:
+ - With curriculum: sort by tool count (easy → hard within epoch)
+ - Without curriculum: random shuffle
+ - Same total data, same epochs
+
+Metrics:
+ - val_ppl convergence curve
+ - Per-tool-count call_f1 at epoch 1, 5, 10
+
+Hypothesis:
+ - Curriculum helps early training (faster initial convergence)
+ - By end of training, gap narrows
+ - Biggest benefit on high-tool-count examples (model learns
+ simple patterns first, then extends to complex cases)
+
+What this shows:
+ - Curriculum is useful but not critical
+ - Justifies the easy→hard ordering for tool-call training
+
+
+### Exp 9: ZCRMSNorm vs Standard RMSNorm [NICE-TO-HAVE]
+
+Setup:
+ - ZCRMSNorm (γ init to 0, applied as (1+γ)*x/RMS)
+ - Standard RMSNorm (γ init to 1, applied as γ*x/RMS)
+ - Same architecture, same recipe
+
+Metrics:
+ - Training loss curves (first 1000 steps: stability)
+ - Final val_ppl and call_f1
+ - Gradient norm trajectory
+
+Hypothesis:
+ - ZCRMSNorm has smoother early training (no initial activation spikes)
+ - Both converge to similar final quality
+ - ZCRMSNorm shows lower gradient norm variance in first 500 steps
+
+What this shows:
+ - Zero-centering helps training stability, especially at small scale
+ - Free improvement: no added cost, just different initialization
+
+
+### Priority
+
+Must-have (core claims):
+ 1. Enc-dec vs dec-only (Exp 1) — justifies architecture choice
+ 2. Full recipe ablation (Exp 2) — proves co-design
+ 3. Matryoshka Pareto (Exp 3) — proves elastic inference
+
+Strong additions:
+ 4. Optimizer comparison (Exp 4) — justifies Muon+Adam
+ 5. QAT + pruning compression (Exp 5) — deployment story
+ 6. Activation comparison (Exp 6) — justifies DReLU
+ 7. Loss weight sensitivity (Exp 7) — justifies structured weighting
+
+Nice-to-have:
+ 8. Curriculum effect (Exp 8) — training efficiency
+ 9. ZCRMSNorm ablation (Exp 9) — initialization choice
diff --git a/notes/sparsification.txt b/notes/sparsification.txt
new file mode 100644
index 0000000..dbf6ef4
--- /dev/null
+++ b/notes/sparsification.txt
@@ -0,0 +1,521 @@
+Group-Aligned Sparsification for Quantized Edge Deployment
+===========================================================
+
+## 1. Problem
+
+Needle targets tiny edge devices (phones, microcontrollers, wearables)
+where memory, compute, and battery are constrained. Two standard
+compression techniques exist independently:
+
+ (a) Quantization: reduce weight precision (FP16 → INT4), 4x memory savings
+ (b) Pruning: zero out weights, skip multiplications, save compute + energy
+
+The problem: these are typically applied independently, with mismatched
+granularity. Standard unstructured pruning zeros individual weights,
+but quantization operates on groups (32 consecutive weights share one
+scale factor). A group with 31 zeros and 1 nonzero still needs its
+full scale factor, full memory, and full compute — the sparsity is
+invisible to the quantization kernel.
+
+
+## 2. Key Insight: Align Pruning Groups with Quantization Groups
+
+If pruning and quantization use the same group structure, an entire
+pruned group becomes a contiguous block of zeros that hardware can
+skip completely:
+
+ Weight matrix W: (in_features, out_features)
+
+ Group-aligned view (group_size=32):
+ ┌────────────────────────────────────────────┐
+ │ group 0: [w_0, w_1, ..., w_31] scale_0 │ ← LIVE (has nonzeros)
+ │ group 1: [0, 0, ..., 0] scale_1 │ ← DEAD (all zeros)
+ │ group 2: [w_64, w_65, ..., w_95] scale_2 │ ← LIVE
+ │ group 3: [0, 0, ..., 0] scale_3 │ ← DEAD
+ │ ... │
+ └────────────────────────────────────────────┘
+
+ Unstructured sparsity (misaligned):
+ ┌────────────────────────────────────────────┐
+ │ group 0: [w_0, 0, w_2, 0, ..., 0, w_31] │ ← half zeros, but
+ │ scale_0 still needed │ can't skip group
+ │ all 32 multiplications still run │
+ └────────────────────────────────────────────┘
+
+ Group-aligned sparsity:
+ ┌────────────────────────────────────────────┐
+ │ group 1: [0, 0, 0, 0, ..., 0, 0] │ ← entire group zero
+ │ scale_1 can be skipped │ skip 32 MACs
+ │ skip memory load for 32 weights │ skip scale load
+ │ skip 32 multiplications │ zero energy
+ └────────────────────────────────────────────┘
+
+When a full group is zero:
+ - Skip loading 32 INT4 weights from memory (16 bytes saved per group)
+ - Skip loading the group scale factor (2 bytes saved)
+ - Skip 32 multiply-accumulate operations
+ - Skip the dequantization step for that group
+ - On battery-powered devices: fewer memory accesses = less energy
+
+The key: our QAT and pruning both use group_size=32. This is not
+coincidence — they are co-designed to produce hardware-friendly
+compressed models.
+
+
+## 3. Architecture
+
+Three components: group-wise magnitude scoring, cubic gradual schedule,
+and mask application.
+
+### 3a. Group-Wise Magnitude Scoring
+
+For each weight matrix, divide the input dimension into groups of 32:
+
+ W ∈ R^{in × out} → W_grouped ∈ R^{num_groups × 32 × out}
+
+ Score each group by the L1 norm of its elements:
+
+ score_g = Σ_{i=0}^{31} Σ_{j=0}^{out-1} |W_grouped[g, i, j]|
+
+ This gives one score per group. Groups with low total magnitude
+ contribute least to the output and are pruned first.
+
+Global threshold: all group scores across ALL weight matrices are
+collected and sorted. The threshold is set at the s-th percentile,
+where s is the target sparsity. This means pruning is globally
+balanced — a layer with many unimportant groups loses more than a
+layer with all important groups.
+
+ all_scores = concat([scores from every weight matrix])
+ threshold = percentile(all_scores, sparsity * 100)
+ mask[g] = 1 if score[g] > threshold else 0
+
+Binary mask: the mask has the same shape as the weight matrix.
+All 32 weights in a pruned group are zeroed together.
+
+ mask_grouped[g, :, :] = broadcast(score[g] > threshold)
+ mask = mask_grouped.reshape(in_features, out_features)
+
+### 3b. Handling nn.scan Parameters
+
+nn.scan stacks layer parameters as 3D tensors: (num_layers, in, out).
+Each layer is pruned independently — different layers can have different
+sparsity patterns, but the global threshold is shared.
+
+ For W ∈ R^{num_layers × in × out}:
+ for each layer l:
+ W[l] → group → score → collect into global pool
+ global threshold from all scores across all layers and all matrices
+ for each layer l:
+ W[l] → group → mask using global threshold
+
+This ensures pruning pressure is distributed based on actual weight
+magnitude, not layer position.
+
+
+### 3c. Cubic Gradual Schedule
+
+One-shot pruning (remove 50% of groups instantly) causes irrecoverable
+accuracy loss. Instead, gradually increase sparsity following a cubic
+schedule (Zhu & Gupta 2017):
+
+ s(t) = s_final × (1 - (1 - (t - t_start) / (t_end - t_start))³)
+
+ ┌─────────────────────────────────────────────────┐
+ │ sparsity │
+ │ ▲ s_final ─── ─── ─── ─── ─── ──│
+ │ │ ╱ │
+ │ │ ╱ │
+ │ │ ╱ ← fast ramp │
+ │ │ ╱ │
+ │ │ ╱ │
+ │ │ _╱ ← slow start │
+ │ │ __╱ │
+ │ │ ___╱ │
+ │ │ ────╱ │
+ │ 0%─────┬──────────────────┬─────────────→ step │
+ │ t_start (33%) t_end (67%) │
+ └─────────────────────────────────────────────────┘
+
+ Default schedule within pruning epoch:
+ t_start = 33% of epoch steps (first third: no pruning)
+ t_end = 67% of epoch steps (middle third: gradual pruning)
+ remaining third: mask locked, fine-tune surviving weights
+
+ Mask recomputation: every prune_interval=100 steps during the
+ ramp, recompute the mask at the current sparsity level. Between
+ recomputations, the mask is frozen — weights can recover.
+
+Why cubic (not linear):
+ - Slow start: remove only the most obviously unimportant groups first.
+ The model barely notices.
+ - Fast middle: bulk of pruning happens here, while the model has
+ already adapted to losing some groups.
+ - Slow end: final fine-grained pruning. The remaining groups are
+ closer in importance — take care removing the last few.
+
+After pruning epoch completes:
+ - Mask is permanently locked
+ - All subsequent training respects the mask (w = w * mask after each step)
+ - The model fine-tunes only the surviving weights
+
+
+### 3d. Mask Application in Training Loop
+
+The mask is applied after each gradient update, not during:
+
+ # In _train_step:
+ state = state.apply_gradients(grads=grads) # update ALL weights
+ new_params = cond(
+ do_prune,
+ lambda: tree_map(lambda w, m: w * m, state.params, prune_mask),
+ lambda: state.params,
+ )
+ state = state.replace(params=new_params)
+
+This is "mask-after-update" (not "mask-before-gradient"). Both are
+used in practice, but mask-after-update has an advantage: the optimizer
+state (momentum, variance estimates) remains valid for pruned weights.
+If we ever want to "un-prune" during the gradual schedule (a group
+recovers importance), the optimizer can immediately use its momentum —
+no cold start.
+
+In practice, during the cubic ramp, groups can be un-pruned: a group
+zeroed at 20% sparsity might survive at 25% sparsity if the global
+threshold shifts. This is a natural consequence of global thresholding
+with recomputation.
+
+
+## 4. Co-Design with INT4 Quantization
+
+### 4a. Shared Group Size
+
+Both QAT and pruning use group_size=32. This is the critical co-design:
+
+ Quantization groups:
+ W_grouped = W.reshape(num_groups, 32, out_features)
+ scale[g] = max(|W_grouped[g]|) / 7 (one scale per group)
+ W_q[g] = clip(round(W_grouped[g] / scale[g]), -8, 7)
+
+ Pruning groups:
+ W_grouped = W.reshape(num_groups, 32, out_features)
+ score[g] = sum(|W_grouped[g]|)
+ mask[g] = score[g] > threshold
+
+ Same reshape. Same groups. The pruned groups are exactly the groups
+ the quantization kernel operates on.
+
+### 4b. Hardware Execution Model
+
+On edge INT4 kernels (ARM NEON, Apple AMX, RISC-V V):
+
+ For each output element y[j]:
+ for each group g in [0, num_groups):
+ if mask[g] == 0: ← group is pruned
+ continue ← skip entirely: no load, no compute
+ load scale[g] ← 2 bytes
+ load W_q[g, :, j] ← 16 bytes (32 × 4-bit)
+ dequant = W_q * scale ← 32 multiplies
+ y[j] += dot(dequant, x[g*32:(g+1)*32]) ← 32 MACs
+ end
+
+ With 50% group sparsity:
+ - 50% fewer memory loads (weights + scales)
+ - 50% fewer multiply-accumulate operations
+ - ~50% less energy (memory access dominates energy on edge)
+ - Model size: 50% fewer groups × (16 bytes weights + 2 bytes scale)
+ = 50% size reduction ON TOP of 4x from INT4
+
+ Combined compression:
+ FP16 baseline: 100% (2 bytes per weight)
+ INT4 only: 25% (0.5 bytes per weight + scale overhead)
+ INT4 + 50% sparse: ~13% (half the INT4 groups)
+
+ For a 50M parameter model:
+ FP16: 100 MB
+ INT4: 25 MB (+ ~2 MB scales)
+ INT4 + 50% sparse: ~14 MB
+
+
+### 4c. Why Group Size 32
+
+ - ARM NEON: 128-bit SIMD → 32 INT4 values per register
+ - Apple AMX: operates on blocks that are multiples of 32
+ - CUDA: warp size 32, INT4 tensor cores operate on 32-element groups
+ - RISC-V V: typical VLEN=128+ → 32+ INT4 elements per vector register
+
+Group size 32 means each pruned group corresponds to exactly one
+hardware vector operation that can be skipped. Smaller groups (16)
+waste half the register; larger groups (64) are too coarse for
+fine-grained pruning. 32 is the hardware-natural sweet spot.
+
+
+## 5. Comparison with Other Pruning Approaches
+
+### vs. Unstructured Pruning (SparseGPT, Wanda)
+
+ Unstructured: zeros individual weights anywhere in the matrix.
+ 50% unstructured sparsity → every group still has ~half its weights.
+ No group can be skipped. Requires sparse matmul kernels (CSR/CSC),
+ which have poor hardware utilization on edge devices.
+
+ Group-aligned: zeros entire groups. 50% sparsity → 50% of groups
+ are completely empty. Dense matmul kernels skip entire groups.
+ No sparse format overhead. No index storage.
+
+ Accuracy comparison:
+ Unstructured can achieve slightly better accuracy at the same
+ sparsity level (finer granularity → more choices). But the
+ speedup is often zero on real hardware without sparse kernel support.
+
+ Our trade: slightly coarser pruning for guaranteed hardware speedup.
+
+### vs. N:M Structured Sparsity (2:4 on NVIDIA Ampere+)
+
+ N:M sparsity: exactly N zeros in every M consecutive elements.
+ 2:4 sparsity on Ampere/Hopper: hardware-native 2x speedup.
+
+ Limitations:
+ - Only available on NVIDIA Ampere+ GPUs (not edge devices)
+ - Fixed 50% sparsity (can't do 30% or 70%)
+ - Fixed pattern (2:4) — no flexibility
+ - Not compatible with INT4 group quantization grouping
+
+ Our approach: works on any hardware with vector instructions.
+ Variable sparsity (0-90%). Compatible with INT4 group quantization.
+ No special hardware required.
+
+### vs. Channel/Filter Pruning
+
+ Channel pruning: removes entire rows/columns of weight matrices.
+ Maximally structured (every output using that channel is affected).
+
+ Limitation: too coarse. Removing an entire channel of d_model=512
+ is a huge capacity hit. For small models, even removing one channel
+ is noticeable.
+
+ Our group pruning: operates at the granularity of 32 input features
+ per output. Much finer than channel pruning. A matrix can lose
+ some groups in one region while keeping others — the pruning
+ pattern adapts to the weight distribution.
+
+
+## 6. Experiment Design
+
+### Exp 1: Group-Aligned vs Unstructured Sparsity [MUST-HAVE]
+
+Setup:
+ - Same model, same training, three pruning modes:
+ (a) Group-aligned (group_size=32, our approach)
+ (b) Unstructured (individual weight magnitude pruning)
+ (c) No pruning (baseline)
+ - Sparsity levels: 0%, 25%, 50%, 70%, 90%
+ - All with QAT (INT4, group_size=32)
+
+Metrics:
+ | Sparsity | Mode | val_ppl | call_f1 | quant_ppl | actual_speedup |
+ |----------|---------------|---------|---------|-----------|----------------|
+ | 0% | baseline | | | | 1.0x |
+ | 25% | unstructured | | | | |
+ | 25% | group-aligned | | | | |
+ | 50% | unstructured | | | | |
+ | 50% | group-aligned | | | | |
+ | 70% | unstructured | | | | |
+ | 70% | group-aligned | | | | |
+ | 90% | unstructured | | | | |
+ | 90% | group-aligned | | | | |
+
+ actual_speedup: measured on target edge device (ARM CPU with NEON)
+
+Hypothesis:
+ - Unstructured has slightly better accuracy at same sparsity (~1-2%)
+ - Group-aligned has MUCH better actual speedup (near-linear with sparsity)
+ - Unstructured has ~1x speedup (no hardware skip support on edge)
+ - Group-aligned at 50% matches unstructured at 50% accuracy but with
+ real 1.8-2x speedup
+ - The quant_ppl gap between aligned/unstructured is small because QAT
+ adapts to both patterns
+
+What this shows:
+ - Group alignment is essential for actual deployment speedup
+ - The small accuracy cost is worth the massive efficiency gain
+ - Unstructured sparsity is a "paper metric" — looks good, no real speedup
+
+
+### Exp 2: Gradual vs One-Shot Pruning [MUST-HAVE]
+
+Setup:
+ - Same target sparsity (50%), three schedules:
+ (a) Gradual cubic (our approach: 0% → 50% over 33%-67% of epoch)
+ (b) One-shot at epoch start (prune to 50% immediately, then train)
+ (c) One-shot at epoch end (train, then prune to 50%)
+ - Same total training budget
+
+Metrics:
+ | Schedule | val_ppl | call_f1 | name_f1 | value_acc |
+
+ Also: val_ppl convergence curve (how quickly accuracy recovers)
+
+Hypothesis:
+ - Gradual cubic is best (model adapts smoothly)
+ - One-shot at start: worst (massive capacity loss before adaptation)
+ - One-shot at end: middle (full training, then sudden pruning)
+ - Gradual reaches same final quality 30-50% faster than one-shot-start
+
+What this shows:
+ - Gradual pruning is necessary for small models
+ - The cubic schedule is justified (smooth ramp > abrupt cut)
+
+
+### Exp 3: Group Size Sweep [MUST-HAVE]
+
+Setup:
+ - Group sizes: 8, 16, 32, 64, 128
+ - QAT group size always matches pruning group size
+ - 50% target sparsity
+
+Metrics:
+ | group_size | val_ppl | call_f1 | quant_ppl | model_size(MB) | tok/s(edge) |
+ |------------|---------|---------|-----------|----------------|-------------|
+ | 8 | | | | | |
+ | 16 | | | | | |
+ | 32 | | | | | |
+ | 64 | | | | | |
+ | 128 | | | | | |
+
+Hypothesis:
+ - group_size=8: best accuracy (finest granularity), worst speedup
+ (groups don't fill hardware vectors)
+ - group_size=32: sweet spot (matches hardware vector width)
+ - group_size=128: worst accuracy (too coarse), but fastest per-group
+ skip (fewer groups to iterate)
+ - Quantization overhead (scale storage) is lowest at group_size=128
+ but accuracy suffers
+
+What this shows:
+ - group_size=32 is the right choice (hardware-natural + good accuracy)
+ - Justifies the co-design: QAT and pruning both at 32
+
+
+### Exp 4: Combined Compression Frontier [STRONG]
+
+Setup:
+ - Sweep the sparsity-quantization space:
+ FP16 + {0%, 25%, 50%, 70%} sparsity
+ INT8 + {0%, 25%, 50%, 70%} sparsity
+ INT4 + {0%, 25%, 50%, 70%} sparsity
+ - All with QAT (matched to quantization level)
+ - Report model size, inference speed, and accuracy
+
+Metrics:
+ Plot: model_size(MB) vs call_f1 (Pareto frontier)
+ Plot: tok/s(edge) vs call_f1 (Pareto frontier)
+
+ Annotate each point with (quant_level, sparsity%)
+
+Hypothesis:
+ - INT4 + 50% sparse dominates the Pareto frontier for edge
+ - FP16 + 0% is best accuracy but impractical (too large/slow)
+ - INT4 + 0% is a strong baseline but leaving performance on table
+ - INT4 + 70% pushes into extreme compression with acceptable loss
+
+What this shows:
+ - Quantization and sparsification are complementary (multiplicative compression)
+ - The co-designed approach (shared group_size) maximizes both
+ - Practical deployment recipe: INT4 + 50% sparse ≈ 8x compression
+ with <5% accuracy loss
+
+
+### Exp 5: Energy Measurement on Edge Device [STRONG]
+
+Setup:
+ - Deploy on target edge device (e.g. Raspberry Pi 5, Pixel phone)
+ - Measure per-inference energy (joules) using hardware power monitor
+ - Compare: FP16 baseline, INT4-only, INT4+50% group-sparse
+
+Metrics:
+ | Condition | Energy(mJ) | Latency(ms) | call_f1 | Model size |
+ |-------------------|------------|-------------|---------|------------|
+ | FP16 baseline | | | | |
+ | INT4 | | | | |
+ | INT4 + 50% sparse | | | | |
+
+ Also: energy breakdown by component (memory load, compute, other)
+
+Hypothesis:
+ - Memory loads dominate energy on edge (>60% of total)
+ - INT4 + 50% sparse reduces memory energy by ~87% (4x quant × 2x sparse)
+ - Compute energy scales linearly with sparsity (skip zero groups)
+ - Total energy reduction: 5-8x vs FP16 baseline
+
+What this shows:
+ - Group-aligned sparsity directly maps to energy savings
+ - This is the deployment story: same model, 5-8x less energy per inference
+ - Battery life argument for edge devices
+
+
+### Exp 6: Pruning Schedule Ablation [STRONG]
+
+Setup:
+ - Vary pruning schedule parameters:
+ start_frac: {0.1, 0.2, 0.33, 0.5}
+ end_frac: {0.5, 0.67, 0.8, 0.9}
+ interval: {10, 50, 100, 500}
+ - 50% target sparsity
+
+Metrics:
+ - Final call_f1 for each combination
+ - Training efficiency: steps to reach 90% of final accuracy
+
+Hypothesis:
+ - start_frac=0.33, end_frac=0.67 (middle third) is near-optimal
+ - Too early start (0.1): model hasn't learned enough to identify
+ unimportant groups
+ - Too late end (0.9): not enough fine-tuning time after pruning
+ - Interval=100 is sufficient (mask doesn't need per-step updates)
+
+What this shows:
+ - The default schedule is well-chosen
+ - Pruning during the "middle" of training is optimal (model has
+ learned structure but still has time to adapt)
+
+
+### Exp 7: Sparsity Pattern Visualization [NICE-TO-HAVE]
+
+Setup:
+ - Visualize the pruning mask at 50% sparsity
+ - Heat maps: which groups survive in each layer
+ - Compare encoder vs decoder patterns
+
+Presentation:
+ - Matrix visualization: rows = input groups, columns = output features
+ - Color: white = pruned, black = kept
+ - One plot per layer type (encoder self-attn, decoder cross-attn, FFN)
+
+Hypothesis:
+ - FFN layers are more prunable than attention projections
+ - Encoder FFN: earlier layers keep more groups (basic features)
+ - Decoder cross-attention: keeps groups aligned with tool-relevant
+ encoder positions
+ - Symmetric pattern in attention Q/K but asymmetric in V/O
+
+What this shows:
+ - The model learns which groups matter
+ - Visual evidence that global thresholding distributes sparsity
+ intelligently across layers
+
+
+### Priority
+
+Must-have (core claims):
+ 1. Group-aligned vs unstructured (Exp 1) — proves hardware speedup
+ 2. Gradual vs one-shot (Exp 2) — proves schedule necessity
+ 3. Group size sweep (Exp 3) — proves group_size=32 choice
+
+Strong additions:
+ 4. Combined compression frontier (Exp 4) — deployment Pareto
+ 5. Energy measurement (Exp 5) — real-world edge impact
+ 6. Schedule ablation (Exp 6) — hyperparameter justification
+
+Nice-to-have:
+ 7. Sparsity pattern visualization (Exp 7) — interpretability
diff --git a/notes/tool-selection.txt b/notes/tool-selection.txt
new file mode 100644
index 0000000..f00f5dc
--- /dev/null
+++ b/notes/tool-selection.txt
@@ -0,0 +1,322 @@
+Contrastive Tool Selection via Shared Encoder
+==============================================
+
+## 1. Problem
+
+Tool-calling models receive a set of tool definitions in-context and must
+select the right tool(s) for a given query. Two challenges:
+
+ (a) Context window pressure: with 1024 encoder tokens, fitting many
+ tool definitions alongside the query is tight. Real APIs can have
+ 50-200 endpoints — they simply don't fit.
+
+ (b) Selection from large catalogs: even if tools fit, the model must
+ discriminate between semantically similar tools (e.g.,
+ search_flights vs search_hotels vs search_restaurants). This is a
+ retrieval problem, not a generation problem.
+
+The standard approach is a separate retrieval model (e.g., embedding API)
+to pre-filter tools before passing them to the generator. This adds a
+second model, a second inference pass, and a mismatch between the
+representations used for retrieval and generation.
+
+
+## 2. Key Insight: Shared Encoder, Dual Use
+
+Our encoder already processes both queries and tool definitions during
+generation (encoder input = [query | <tools> | tools...]). The same
+encoder representations that attend to tools during generation can be
+repurposed for retrieval by adding a lightweight contrastive head.
+
+Architecture:
+
+ ┌─────────────┐ ┌─────────────┐
+ │ "What is │ │ {"name": │
+ │ the weather│ │ "get_weather│
+ │ in SF?" │ │ "params": │
+ │ │ │ {"location"│
+ └──────┬──────┘ │ : "string"}│
+ │ │ } │
+ ▼ └──────┬──────┘
+ ┌──────────────┐ │
+ │ │ │
+ │ Shared │◄──────────┘
+ │ Encoder │
+ │ (8 layers) │
+ │ │
+ └──────┬───────┘
+ │
+ ┌────┴────┐
+ │ │
+ ▼ ▼
+ ┌─────┐ ┌─────────┐
+ │Mean │ │Contras- │
+ │Pool │ │tive │
+ │ │ │Proj │
+ └──┬──┘ │(d→128) │
+ │ └────┬────┘
+ │ │
+ ▼ ▼
+ (d_model) (128-d, L2-normalized)
+ │ │
+ ▼ ▼
+ Cross- Cosine similarity
+ attention for retrieval
+ in decoder
+
+The contrastive projection head is a single linear layer (d_model → 128)
+followed by L2 normalization. It's trained with CLIP-style symmetric
+cross-entropy loss: in each batch, the B queries are paired with B
+positive tools, and the B²-B in-batch negatives provide contrastive signal.
+
+This means:
+ - No separate retrieval model — same encoder, same parameters
+ - Retrieval representations are grounded in the same space used for
+ generation — what the encoder learns for cross-attention also helps
+ distinguish tools
+ - The contrastive head adds <0.01% parameters (one 512×128 matrix)
+
+
+## 3. Training Details
+
+CLIP-style loss with learnable temperature:
+
+ loss = 0.5 * (CE(q·t^T/τ, labels) + CE(t·q^T/τ, labels))
+
+ - q, t ∈ R^{B×128}, L2-normalized
+ - τ = exp(log_temp), clamped to [1/100, 100]
+ - labels = arange(B) (diagonal is positive)
+ - In-batch negatives: B-1 negatives per query, free
+
+Training schedule:
+ - Contrastive loss weighted at 0.1× text loss
+ - Computed every 100 steps (not every step — saves compute, contrastive
+ signal is less noisy so doesn't need per-step updates)
+ - Uses the same encoder parameters as text generation (shared, not frozen)
+
+Data construction:
+ - Each training example has a query, a list of tools, and the answer
+ (which tools were called)
+ - For contrastive training: extract (query, positive_tool) pairs where
+ positive_tool is any tool that was actually called
+ - Each query may have multiple positives; one is randomly sampled per batch
+ - Individual tools are tokenized separately (max 256 tokens each)
+
+
+## 4. Inference Pipeline
+
+Two-stage inference for large catalogs:
+
+ Stage 1 — Retrieval:
+ encode_contrastive(query) → q_emb (128-d)
+ encode_contrastive(tools) → t_embs (N×128)
+ scores = q_emb @ t_embs.T
+ top_k = argsort(-scores)[:k]
+
+ Stage 2 — Generation:
+ encoder_input = [query | <tools> | top_k_tools]
+ decoder generates tool calls with only the selected tools in context
+
+This naturally handles the context window constraint: even with 200
+available tools, we retrieve top-5 and fit them easily in 1024 tokens.
+The retrieval and generation use the same encoder weights, so what the
+retrieval stage selects is precisely what the generation stage can
+attend to effectively.
+
+For small catalogs (≤5 tools): skip retrieval, pass all tools directly.
+The contrastive head is never used at inference in this case — it was
+only useful as a training regularizer (the shared representations
+benefit generation even without explicit retrieval).
+
+
+## 5. Why This Design
+
+### vs. Separate retrieval model (e.g., embed API + generator)
+ - One model instead of two → simpler deployment, less memory
+ - Shared representations → retrieval is aligned with generation
+ - No train/inference distribution shift between retriever and generator
+
+### vs. Stuffing all tools in context
+ - Doesn't scale: 50 tools × ~50 tokens each = 2500 tokens, exceeds
+ 1024 encoder context
+ - Even if they fit, attention over many irrelevant tools dilutes signal
+ - Retrieval pre-filtering gives the generator a clean, focused context
+
+### vs. Fine-tuning a separate contrastive model
+ - The encoder is already "seeing" tools during generation training
+ - Adding a projection head extracts what it already knows
+ - Multi-task learning: contrastive loss acts as regularizer for the
+ encoder, improving generation even on in-context tools
+
+### The regularization effect
+ Even when retrieval isn't used at inference (small catalogs), the
+ contrastive objective during training forces the encoder to build
+ discriminative representations of tools. This helps cross-attention
+ in the decoder: the encoder outputs for each tool are more distinct,
+ making it easier for the decoder to attend to the right tool.
+
+
+## 6. Experiment Design
+
+### Exp 1: Retrieval Quality [MUST-HAVE]
+
+Setup:
+ - Validation set, each example has query + tools + ground truth calls
+ - Encode query and all tools from that example contrastively
+ - Rank tools by cosine similarity, check if positive tools are in top-k
+
+Metrics:
+ | k | Recall@k |
+ |-----|----------|
+ | 1 | |
+ | 3 | |
+ | 5 | |
+ | 10 | |
+
+ Also report MRR (mean reciprocal rank of first positive).
+
+Stratify by:
+ - Number of tools per example (harder with more tools)
+ - Number of positive tools (multi-call examples)
+
+What this shows:
+ - The shared encoder produces discriminative tool representations
+ - Recall@5 should be very high (>95%) — retrieval is nearly solved
+
+
+### Exp 2: Ablation — Generation Quality With vs Without Contrastive [MUST-HAVE]
+
+Setup:
+ - Train two models: with contrastive loss (λ=0.1) and without (λ=0)
+ - Same architecture, same data, same hyperparameters otherwise
+ - Evaluate generation metrics on val set (no retrieval at inference)
+
+Metrics:
+ | Condition | name_f1 | call_f1 | exact_match | value_acc |
+
+Hypothesis:
+ - Even without using retrieval at inference, the contrastive-trained
+ model should have better name_f1 and call_f1
+ - The regularization effect: contrastive loss forces discriminative
+ encoder representations → better cross-attention → better generation
+
+What this shows:
+ - Contrastive learning isn't just for retrieval — it improves generation
+ - This is the key insight: one objective helps two tasks
+
+
+### Exp 3: Retrieve-then-Generate vs Full Context [MUST-HAVE]
+
+Setup:
+ - Create synthetic scenarios with large tool catalogs (20, 50, 100 tools)
+ by combining tools from multiple validation examples
+ - Three conditions:
+ (a) All tools in context (may truncate)
+ (b) Top-5 retrieved tools in context
+ (c) Oracle top-5 (ground truth positive tools + random negatives)
+
+Metrics:
+ | Condition | 20 tools | 50 tools | 100 tools |
+ | | name_f1 call_f1| name_f1 call_f1| name_f1 call_f1|
+ |----------------|---------------|---------------|---------------|
+ | All in context | | | |
+ | Retrieved top-5| | | |
+ | Oracle top-5 | | | |
+
+Hypothesis:
+ - With 20 tools: "all in context" works but retrieved is comparable
+ - With 50+ tools: "all in context" degrades (truncation, attention dilution)
+ - Retrieved top-5 stays close to oracle → retrieval solves the scaling problem
+
+What this shows:
+ - Retrieval is necessary for real-world tool catalogs
+ - The gap between retrieved and oracle quantifies retrieval errors
+ - The gap between all-context and retrieved quantifies context pressure
+
+
+### Exp 4: Contrastive Loss Weight Sweep [STRONG]
+
+Setup:
+ - Train models with λ ∈ {0, 0.01, 0.05, 0.1, 0.2, 0.5}
+ - Evaluate both retrieval (Recall@k) and generation (call_f1)
+
+Metrics (line plot, x=λ):
+ - Recall@5 (retrieval quality)
+ - call_f1 (generation quality)
+
+Hypothesis:
+ - λ=0: no retrieval capability, generation baseline
+ - λ=0.01-0.1: retrieval improves rapidly, generation also improves
+ - λ>0.2: retrieval saturates, generation may degrade (contrastive
+ objective starts dominating, encoder optimizes for discrimination
+ over representation)
+
+What this shows:
+ - Sweet spot exists where both tasks benefit
+ - Justifies λ=0.1 as default
+
+
+### Exp 5: Embedding Space Visualization [STRONG]
+
+Setup:
+ - Encode 500 queries and their associated tools
+ - Project to 2D via t-SNE or UMAP
+ - Color by: tool category / query intent / positive vs negative
+
+Presentation:
+ - Scatter plot showing query-tool alignment
+ - Highlight clusters where similar tools group together
+ - Show queries landing near their positive tools
+
+What this shows:
+ - Visual evidence that the shared space is semantically meaningful
+ - Queries and their positive tools are close; negatives are far
+ - Tool clusters correspond to functional categories
+
+
+### Exp 6: Retrieval Latency [STRONG]
+
+Setup:
+ - Measure time to encode and retrieve from catalogs of varying size
+ - Compare: our model's encode_contrastive vs. a separate embedding API
+
+Metrics:
+ | Catalog size | Encode time | Retrieval time | Total |
+
+What this shows:
+ - Retrieval adds negligible latency (single encoder pass + matmul)
+ - Amortization: tool embeddings can be precomputed and cached
+ - No network round-trip to a separate embedding service
+
+
+### Exp 7: Contrastive Frequency Ablation [NICE-TO-HAVE]
+
+Setup:
+ - Vary how often contrastive loss is computed: every step, every 10,
+ every 100 (current), every 500
+ - Same total training budget
+
+Metrics:
+ - Recall@5 convergence curve
+ - Final call_f1
+
+What this shows:
+ - Contrastive signal is low-noise, doesn't need per-step updates
+ - Every-100 is sufficient (saves ~50% of contrastive compute)
+ - Justifies the design choice
+
+
+### Priority
+
+Must-have (core claims):
+ 1. Retrieval quality (Exp 1) — proves it works
+ 2. Ablation: with/without contrastive (Exp 2) — proves dual benefit
+ 3. Retrieve-then-generate scaling (Exp 3) — proves necessity
+
+Strong additions:
+ 4. Loss weight sweep (Exp 4) — justifies hyperparameter
+ 5. Embedding visualization (Exp 5) — intuition and visual appeal
+ 6. Retrieval latency (Exp 6) — practical deployment story
+
+Nice-to-have:
+ 7. Contrastive frequency ablation (Exp 7) — training efficiency
diff --git a/notes/tries.txt b/notes/tries.txt
new file mode 100644
index 0000000..7dfed76
--- /dev/null
+++ b/notes/tries.txt
@@ -0,0 +1,267 @@
+Selective Trie-Constrained Decoding for Tool-Call Generation
+=============================================================
+
+## 1. Problem
+
+The model generates JSON tool calls autoregressively. While it learns
+JSON structure well (99-100% parse rate), it can still hallucinate
+invalid function names or argument keys. The valid tool definitions are
+already in the encoder input — we can constrain the decoder to only
+produce valid names/keys at inference time.
+
+
+## 2. Key Insight: Constrain What's Broken, Not What Works
+
+The remaining errors are semantic, not syntactic:
+
+ Error type Constrainable? Frequency
+ ───────────────────────── ────────────── ─────────
+ Wrong/hallucinated name YES (trie) Common
+ Hallucinated param key YES (trie) Common
+ Wrong argument values NO (open-ended) Common
+ Missing arguments NO Moderate
+ Malformed JSON NO (already 99%) Rare
+
+Full grammar approaches (Outlines, llama.cpp GBNF, guidance) enforce
+valid JSON on every token. This adds 10-30% overhead to solve a problem
+that doesn't exist. Our approach only constrains the ~5% of tokens that
+are inside name/key spans.
+
+
+## 3. Architecture
+
+Three components: a character-level trie, a JSON state machine, and
+a token masking layer.
+
+### 3a. Trie
+
+Given tools: get_weather(location, unit), get_stock(symbol)
+
+Name trie: Param trie (get_weather):
+
+ (root) (root)
+ | / \
+ g l u
+ | | |
+ e o n
+ | | |
+ t c i
+ | | |
+ _ a t*
+ / \ |
+ w s t
+ | | |
+ e t i
+ | | |
+ a o o
+ | | |
+ t c n*
+ | |
+ h k* (* = terminal node)
+ |
+ e
+ |
+ r*
+
+One name trie shared across all tools. One param trie per function,
+built from its parameter keys.
+
+### 3b. JSON State Machine
+
+Tracks position in output to know when to constrain:
+
+ ┌──────┐ "name":" ┌─────────┐ closing " ┌──────┐
+ │ FREE │────────────→│ IN_NAME │────────────→│ FREE │
+ └──────┘ └─────────┘ └──────┘
+ │ │
+ │ "arguments":{ then {" or ," │
+ │ ┌────────────┐ closing " │
+ └───────────────→│ IN_ARG_KEY │─────────────→┘
+ └────────────┘
+
+State transitions driven by suffix matching on generated text.
+Tracks nesting depth to avoid false triggers inside nested values.
+Tracks string values with escape handling to avoid false triggers
+inside string content.
+
+### 3c. Token Masking
+
+At each constrained step:
+
+ 1. Look up current trie node (root + constrained_buf prefix)
+ 2. Gather valid first chars from trie children (+ closing " if terminal)
+ 3. For each candidate token (via TokenIndex first-char lookup):
+ - Walk trie char-by-char through token text
+ - If token contains ", check trie node is terminal at that point
+ - Mark as valid or invalid
+ 4. Set invalid token logits to -inf
+ 5. Fallback: if ALL tokens masked, revert to unconstrained + log warning
+
+Example (IN_NAME state, after consuming "get_"):
+
+ Trie node at "get_" has children: {w, s}
+ Scan TokenIndex for tokens starting with 'w' or 's':
+ "weather" → walk trie: w→e→a→t→h→e→r, all valid → ALLOW
+ "we" → walk trie: w→e, valid prefix → ALLOW
+ "stock" → walk trie: s→t→o→c→k, terminal → ALLOW
+ "sun" → walk trie: s→u, 'u' not child of 's' → BLOCK
+
+Multi-char token boundary handling:
+ Token "ther\"" (closing quote mid-token):
+ Walk trie for 't','h','e','r' → valid path
+ Hit '"' → check if node is terminal → yes (weather*) → ALLOW
+ Correctly handles BPE tokens that span the name boundary.
+
+
+## 4. Feedback Effect
+
+The corrected token feeds back into the decoder buffer. All subsequent
+generation conditions on the valid name:
+
+ Without constraint: ..."name":"get_weathr","arguments":{"locaton":...
+ ~~~~~~~~~~~~ wrong ~~~~~ ~~~~~~~~ wrong ~~~~
+
+ With constraint: ..."name":"get_weather","arguments":{"location":...
+ ~~~~~~~~~~~~ forced ~~~~ ~~~~~~~~~ model sees
+ correct name, generates
+ better arguments
+
+This is why in-place correction beats post-processing — the decoder
+sees correct context when generating arguments.
+
+
+## 5. Comparison with Full Grammar Approaches
+
+ ┌─────────────────────────────────────────────────────────┐
+ │ Full grammar (CFG/PDA) │
+ │ │
+ │ Constrains EVERY token: brackets, colons, commas, │
+ │ string delimiters, value types, nesting, whitespace. │
+ │ Guarantees syntactically valid JSON always. │
+ │ │
+ │ Cost: FSM/PDA transition on every token, large state │
+ │ space, precomputed token masks per state, high memory. │
+ │ Typical overhead: 10-30% slowdown on all tokens. │
+ └─────────────────────────────────────────────────────────┘
+
+ ┌─────────────────────────────────────────────────────────┐
+ │ Selective constraint (trie + state machine) │
+ │ │
+ │ Only constrains tokens inside "name":"___" and │
+ │ argument key "___" spans. Everything else │
+ │ (structure, values, punctuation) is unconstrained. │
+ │ │
+ │ Cost: lightweight state machine tracks JSON position │
+ │ via suffix matching. Trie walk + token masking only │
+ │ activates during name/key spans (~5% of tokens). │
+ │ Zero overhead on remaining ~95% of tokens. │
+ └─────────────────────────────────────────────────────────┘
+
+Why full grammar is overkill here:
+ - 99-100% JSON parse rate means structure is already learned
+ - Full grammar pays per-token cost for ~0% parse improvement
+ - Can force the model into corners where the most likely valid
+ continuation is nonsensical (distorts value generation)
+
+Why selective works:
+ - Targets the actual failure mode (hallucinated names/keys)
+ - Near-zero cost (only active on ~5% of tokens)
+ - Never distorts value generation
+ - Corrected tokens improve downstream generation via feedback
+
+
+## 6. Implementation Details
+
+ token_strings / TokenIndex: cached per tokenizer (built once)
+ Device-to-host transfer: only when state machine is active
+ (IN_NAME or IN_ARG_KEY); FREE state stays on jnp.argmax
+ Enabled by default; disable with --no-constrained
+
+
+## 7. Experiment Design
+
+All trie experiments use our val set (5000 samples) — clean controlled
+variable where only the decoding mode changes. Exact reference calls
+give precise name_f1, param_halluc, value_acc. Per-tool-count breakdown
+already implemented. Reproducible with a seed.
+
+### Exp 1: Ablation — Constrained vs Unconstrained [MUST-HAVE]
+
+Same checkpoint, same val set, three conditions:
+ (a) Unconstrained (--no-constrained)
+ (b) Name-only constraint (trie on function names, keys unconstrained)
+ (c) Full selective constraint (trie on names + argument keys)
+
+ | Condition | name_f1 | call_f1 | exact_match | param_halluc | param_miss | value_acc | json_parse |
+
+Expects: name_f1 improves (a→b), param_halluc drops (b→c), json_parse
+unchanged across all three (confirms structure isn't the problem).
+Needs a "names_only" mode flag in ConstrainedDecoder.
+
+### Exp 2: Latency Overhead [MUST-HAVE]
+
+500 samples, wall-clock time and tok/s, three conditions, 3 runs each.
+
+ | Condition | tok/s | Overhead |
+
+Also report: % tokens where constraint was active, avg time in
+apply_constraints per constrained token.
+Expects: near-zero overhead because constraints activate on ~5% of tokens.
+
+### Exp 3: Error Analysis — What Constraints Fix [MUST-HAVE]
+
+Run constrained and unconstrained on same 1000 samples. Categorize:
+ (a) Both correct → constraint irrelevant
+ (b) Unconstrained wrong, constrained correct → constraint helped
+ (c) Both wrong (same error) → constraint irrelevant
+ (d) Both wrong (different error) → constraint changed but didn't fix
+ (e) Unconstrained correct, constrained wrong → constraint hurt (should be ~0)
+
+For category (b), sub-classify:
+ - Misspelled function name (get_weathr → get_weather)
+ - Hallucinated function name (fetch_weather → get_weather)
+ - Hallucinated parameter key (city → location)
+ - Misspelled parameter key (locaton → location)
+
+Present as sankey diagram + table of top-10 fixes with before/after.
+Category (e) should be empty — proves constraints are safe.
+
+### Exp 4: Breakdown by Tool Count [STRONG]
+
+Use existing per_count breakdown (1-10 tools per example).
+Grouped bar chart: name_f1 and param_halluc per bucket, constrained vs not.
+
+Expects: more tools → more names to confuse → bigger constraint benefit.
+Shows constraint value scales with schema complexity.
+
+### Exp 5: Downstream Effect — Name Correction Improves Arguments [STRONG]
+
+Focus on examples where unconstrained gets the name wrong.
+Compare value_acc and args_acc when name is wrong vs corrected.
+
+Expects: corrected name → better arguments (feedback effect).
+Justifies in-place correction over post-processing.
+
+### Exp 6: Benefit Across Training Epochs [STRONG]
+
+Checkpoints at epochs 1, 3, 5, 7, 10. Each evaluated with and without.
+Line plot: call_f1 constrained vs unconstrained across epochs.
+
+Expects: biggest delta mid-training ("almost right" phase), positive
+everywhere. Shows constraints are complementary to training.
+
+### Exp 7: Full Grammar Comparison [NICE-TO-HAVE]
+
+Implement Outlines-style full JSON grammar constraint as baseline.
+Three-way comparison: unconstrained, full grammar, selective trie.
+
+Expects: full grammar has ~same call_f1 but much higher overhead.
+Strongest competitive claim but most implementation work.
+
+### Exp 8: Out-of-Distribution Tool Schemas [NICE-TO-HAVE]
+
+Synthetic tools with names/keys not in training data (20-50 examples).
+Constrained vs unconstrained.
+
+Expects: OOD names have highest hallucination rate → biggest delta.
+Shows value for custom/long-tail APIs. Needs manual data creation.