Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
add notes
HenryNdubuakuee916036a474
diff --git a/notes/architecture.txt b/notes/architecture.txtnew file mode 100644index 0000000..c421403--- /dev/null+++ b/notes/architecture.txt@@ -0,0 +1,720 @@+Co-Designed Architecture & Training Recipe for Small-Scale Tool Calling+=======================================================================++## 1. Thesis++No single component is novel in isolation. The contribution is a+co-designed system: an encoder-decoder architecture trained with a+multi-objective recipe (generation + contrastive + Matryoshka ++structured weighting + QAT + pruning) that produces a small model+competitive with much larger decoder-only models on tool calling.++The architecture choices are interdependent:+ - Encoder-decoder enables shared-encoder retrieval (see tool-selection.txt)+ - Matryoshka FFN enables elastic deployment at multiple compute budgets+ - Structured loss weighting targets the actual error distribution+ - QAT + pruning enable deployment on edge devices+ - Constrained decoding exploits known schemas (see tries.txt)++Each piece alone is known. The system is the contribution.+++## 2. Architecture++### 2a. Why Encoder-Decoder (Not Decoder-Only)++Almost all tool-calling models are decoder-only. We use encoder-decoder+because tool calling has a natural asymmetry:++ Input (encode once): Output (generate):+ ┌──────────────────────┐ ┌──────────────────────┐+ │ Query: "What's the │ │[{"name":"get_weather"│+ │ weather in SF?" │ │ "arguments": │+ │ <tools> │ │ {"location":"SF"}} │+ │ [{"name":"get_weather│ │ ] │+ │ "parameters":{...}}│ └──────────────────────┘+ │ {"name":"get_stock" │+ │ "parameters":{...}}│+ │ ] │+ └──────────────────────┘++ Encoder input = context (query + tool definitions) — processed once+ Decoder output = structured generation (JSON tool calls) — autoregressive++Benefits:+ (a) The encoder processes all tools in parallel via bidirectional+ self-attention. A decoder-only model processes tools left-to-right,+ which means later tools attend to earlier tools but not vice versa.+ For discriminating between semantically similar tools (search_flights+ vs search_hotels), bidirectional context helps.++ (b) Encoder output is reused across all decoder steps via cross-attention.+ In a decoder-only model, KV cache stores the full context, but the+ representation is computed left-to-right with causal masking — no+ opportunity for the context to self-refine bidirectionally.++ (c) The encoder can serve double duty: cross-attention for generation+ AND contrastive retrieval for tool selection (see tool-selection.txt).+ A decoder-only model would need a separate retrieval module.++ (d) Smaller parameter budget. With the same d_model and layer count,+ an encoder-decoder can encode context more efficiently because+ the encoder layers don't need causal masking, cross-attention+ projections, or autoregressive logic.+++### 2b. Zero-Centered RMSNorm (ZCRMSNorm)++Standard RMSNorm: γ * x / RMS(x), with γ initialized to 1.+Our ZCRMSNorm: (1 + γ) * x / RMS(x), with γ initialized to 0.++ class ZCRMSNorm(nn.Module):+ def __call__(self, x):+ scale = self.param("scale", zeros, (x.shape[-1],))+ rms = sqrt(mean(x^2, axis=-1, keepdims=True) + eps)+ return (1 + scale) * x / rms++At initialization, (1 + 0) * x / RMS(x) = x / RMS(x) — the layer+normalizes but applies no learned scaling. This is similar in spirit+to ReZero (Bachlechner et al. 2020) and fixup initialization: at+init, each sub-layer's contribution is minimal, and the residual+stream dominates. The network starts as a near-identity function+and gradually learns to use each layer.++Why this matters for small models:+ - Small models are more sensitive to initialization. Standard RMSNorm+ with γ=1 can cause early training instability because pre-norm+ layers immediately rescale activations.+ - Zero-init means gradients flow cleanly through the residual stream+ at the start of training, avoiding the "early chaos" period.++Applied everywhere: encoder pre-norm, decoder pre-norm, QK-norm,+ConvModule norm. Consistent zero-centering throughout.+++### 2c. Grouped Query Attention (GQA) with QK-Norm++Standard multi-head attention with grouped KV heads:++ Q: (B, num_heads, T, head_dim) — full heads+ K: (B, num_kv_heads, T, head_dim) — grouped (repeated for attention)+ V: (B, num_kv_heads, T, head_dim) — grouped++ num_heads=8, num_kv_heads=2 → 4:1 ratio, 75% KV parameter savings++QK-norm: apply ZCRMSNorm to Q and K after projection, before attention.++ q = ZCRMSNorm(q_proj(x))+ k = ZCRMSNorm(k_proj(x))+ attn = softmax(q @ k^T / sqrt(d_head))++QK-norm prevents attention logit growth as training progresses (the+"logit drift" problem observed in large-scale training). With zero-init+normalization, Q and K start with unit-scale dot products and gradually+learn scale — stable from step 1.++This combination (GQA + QK-norm + ZCRMSNorm) is individually known+(Gemma 2, ViT-22B use QK-norm; GQA from Ainslie et al. 2023) but the+zero-centered variant on Q/K is our specific choice.+++### 2d. DReLU Activation++Three activation options: SwiGLU, GeGLU, DReLU. Default is DReLU.++ SwiGLU: silu(gate) * up — standard, smooth+ GeGLU: gelu(gate) * up — smoother, slightly more expensive+ DReLU: relu(gate) * relu(up) — simplest, sparsest++ h = relu(W_gate @ x) * relu(W_up @ x)+ out = W_down @ h++DReLU applies ReLU to both gate and up projections independently.+This produces the sparsest activations: any input where either gate+OR up projection is negative gets zeroed. Empirically, ~60-70% of+hidden units are zero at any given step.++Why DReLU for tool calling:+ - Activation sparsity enables hardware-level speedups (skip zero+ multiplications). Critical for edge deployment.+ - Works synergistically with magnitude pruning: sparse activations+ mean many weights have low magnitude → cleaner pruning signal.+ - Cheaper than SwiGLU/GeGLU: no sigmoid/erf computation.+ - For structured output (JSON), the model doesn't need smooth+ gradients for nuanced language — it needs sharp, discriminative+ features for selecting the right tool name and argument keys.+++### 2e. Matryoshka FFN++Train a single model that contains multiple sub-models at different+FFN widths. At inference, choose the width based on compute budget.++ Full model: d_ff = 2048 (all neurons active)+ 2x sub-model: d_ff = 1024 (first 1024 neurons)+ 4x sub-model: d_ff = 512 (first 512 neurons)++Implementation: prefix masking on FFN hidden dimension.++ def __call__(self, x, ffn_mask=None):+ gate = gate_proj(x) # (B, T, d_ff)+ up = up_proj(x) # (B, T, d_ff)+ h = relu(gate) * relu(up)+ if ffn_mask is not None:+ h = h * ffn_mask # zero out neurons beyond width+ return down_proj(h)++During training, each batch is split across widths:++ Batch (B=32):+ items 0-10: full width (d_ff=2048) — all neurons+ items 11-21: 2x width (d_ff=1024) — first half+ items 22-31: 4x width (d_ff=512) — first quarter++ All items share the same forward pass (single jit-compiled function).+ The ffn_mask varies per item. No separate models, no distillation.++The prefix constraint is key: sub-models always use the FIRST k neurons.+This means the first 512 neurons must be useful on their own, the first+1024 must be useful, etc. The training objective forces the model to+organize information by importance along the FFN dimension — most+important features in early neurons.++At inference:+ - Edge device: use 4x sub-model (512 neurons, ~25% FFN compute)+ - Server: use full model (2048 neurons, full quality)+ - Same checkpoint, same parameters, different masks++Why not distillation:+ - Distillation requires training N separate models+ - Matryoshka trains one model, gets N sub-models+ - Sub-models share attention weights — only FFN width changes+ - Parameter estimation for any sub-model is straightforward+++### 2f. Shared Encoder with Contrastive Head++See tool-selection.txt for full details. Summary:++ Encoder → mean pool → Dense(d_model, 128) → L2 normalize → cosine similarity++ Same encoder serves:+ (a) Cross-attention in decoder (generation)+ (b) Contrastive retrieval (tool selection from large catalogs)++ Adds <0.01% parameters (one 512x128 matrix + learnable temperature).+++### 2g. Conformer Convolution Module (Optional)++When conv_kernel_size > 0, each encoder block adds a convolution module+between self-attention and FFN:++ self-attn → ConvModule → FFN++ ConvModule:+ pointwise up (d → 2d) → GLU gate → depthwise conv (k=31) →+ ZCRMSNorm → pointwise down (d → d)++This is the Conformer (Gulati et al. 2020) pattern. The depthwise+convolution captures local patterns (token n-grams in tool definitions,+local acoustic patterns in speech). Complementary to global self-attention.++Primary motivation: speech pathway benefits significantly from local+convolution. For text-only, conv_kernel_size=0 (disabled).+++### 2h. Speech Pathway++Optional speech input via FastConformer-style 8x subsampling:++ Input: (B, T, 80) mel spectrogram+ Output: (B, T//8, d_model) features fed to shared encoder++ Subsampling (3 stages of stride-2 = 8x total):+ Conv2D(1→C, k=3, s=2) + ReLU+ DW-Conv(C, k=3, s=2) + PW-Conv(C→C) + ReLU+ DW-Conv(C, k=3, s=2) + PW-Conv(C→C) + ReLU+ Reshape + Linear → d_model++ SpecAugment (training only):+ 2 time masks (max 100 frames)+ 1 frequency mask (max 27 bins)++The speech subsystem converts audio to the same representation space+as text. After subsampling, the encoder processes speech features+identically to text embeddings — same attention, same FFN, same+contrastive head. Unified encoder for both modalities.+++### 2i. Memory-Efficient Layer Stacking++Both encoder and decoder use nn.scan with remat:++ ScanBlock = nn.scan(+ nn.remat(BlockBody),+ variable_axes={"params": 0}, # stack params along axis 0+ split_rngs={"params": True},+ length=num_layers,+ )++ - nn.scan: shares the block's Python code across layers, stacking+ parameters as (num_layers, ...) tensors. Reduces compilation time.+ - nn.remat: recomputes activations during backward pass instead of+ storing them. Trades ~30% more compute for ~60% less memory.++For a small model this matters less, but it enables scaling experiments+with deeper models on the same hardware.+++## 3. Training Recipe++### 3a. Muon + Adam Split Optimizer++Not all parameters benefit from the same optimizer. We split:++ Muon optimizer: all Dense kernels (2D+ tensors)+ - Orthogonalize gradients via Newton-Schulz polar decomposition+ - Nesterov momentum (μ=0.95)+ - Weight decay 0.01+ - Separate WSD learning rate schedule++ Adam optimizer: everything else (embeddings, norms, biases, 1D params)+ - AdamW with β2=0.95 (lower than default 0.999 for stability)+ - No weight decay+ - Separate WSD learning rate schedule++Muon's gradient orthogonalization:++ def _newton_schulz(G, steps=5):+ a, b, c = 3.4445, -4.7750, 2.0315+ X = G / norm(G)+ for _ in range(steps):+ A = X @ X^T+ B = b*A + c*A@A+ X = a*X + B@X+ return X++ This approximates the polar decomposition G = U*S*V^T and returns U*V^T.+ The gradient is projected onto the nearest orthogonal matrix.++Why Muon for attention/FFN kernels:+ - Orthogonalized updates prevent weight matrix collapse (rank+ deficiency) in small models where d_model is limited.+ - Newton-Schulz is cheap: 5 iterations of matrix multiply, no SVD.+ - Empirically, Muon converges faster than Adam on projection matrices.++Why Adam for the rest:+ - Embeddings are lookup tables — orthogonalization doesn't apply.+ - Norm scales are 1D — orthogonalization is meaningless.+ - Adam's adaptive learning rates are better for heterogeneous params.+++### 3b. Warmup-Stable-Decay (WSD) Schedule++ ┌─────────────────────────────────────────────────────────┐+ │ LR │+ │ ▲ │+ │ │ ┌──────────────────────────────┐ │+ │ │ /│ stable │\ │+ │ │ / │ │ \ cosine │+ │ │ / │ │ \ decay │+ │ │ / │ │ \ │+ │ │ / │ │ \____ │+ │ │/ warmup │ │+ │ └──────┴──────────────────────────────┴────────→ step │+ │ 0% ~5% 85% 100% │+ └─────────────────────────────────────────────────────────┘++ - Warmup: linear ramp from 0 to peak (first ~5% of steps)+ - Stable: hold at peak (majority of training)+ - Decay: cosine anneal to 5% of peak (last 15% of steps)++Applied separately to Adam and Muon with different peak values.+WSD is simpler than cosine-with-restarts and matches or beats it+for fixed-budget training (MiniCPM, Zhai et al. 2022).+++### 3c. Quantization-Aware Training (QAT)++Every 100 steps, apply fake INT4 quantization to all Dense kernels:++ def _fake_quantize_int4(w, group_size=32):+ # Group along input dimension+ w_grouped = w.reshape(num_groups, group_size, out_feat)+ # Per-group symmetric quantization+ scale = max(abs(w_grouped), axis=1) / 7.0+ w_q = clip(round(w_grouped / scale), -8, 7) * scale+ # Straight-through estimator+ return w + stop_gradient(w_q - w)++ - Symmetric INT4: 4-bit integers [-8, 7] with per-group scale+ - Group size 32: each 32 consecutive input features share a scale+ - STE: forward uses quantized weights, backward uses real gradients+ - Applied conditionally: only every 100 steps (reduces overhead)++The model learns to be robust to INT4 quantization during training.+At deployment, weights are actually quantized to INT4 → 4x memory+reduction, hardware-accelerated INT4 matmul on supporting devices.++Why every 100 steps instead of every step:+ - QAT adds ~15% overhead per step (extra quantize + forward pass)+ - Every 100 steps: <0.15% amortized overhead+ - Weights change slowly — quantization noise from step N is still+ representative at step N+99+ - Same rationale as contrastive loss frequency (see tool-selection.txt)+++### 3d. Group-Aligned Gradual Pruning++See sparsification.txt for full details. Summary:++ - Group-wise magnitude pruning with group_size=32 — same groups as QAT+ - Cubic gradual schedule: 0% → target over middle third of epoch+ - Global threshold across all layers and matrices+ - Mask applied after gradient update, locked after pruning epoch++ The critical co-design: pruned groups are exactly the groups the INT4+ quantization kernel operates on. A pruned group = zero weights + zero+ scale factor = zero memory loads + zero MACs. On edge INT4 kernels,+ 50% group sparsity → 50% fewer memory accesses and 50% fewer compute+ operations, directly translating to battery savings.+++### 3e. Structured Per-Token Loss Weighting++Not all tokens in the decoder output are equally important:++ [{"name":"get_weather","arguments":{"location":"San Francisco"}}]++ Token role Weight Rationale+ ────────── ────── ─────────+ name value 3.0 Wrong name = completely wrong call+ arg value 2.0 Wrong value = wrong behavior+ arg key 1.5 Wrong key = hallucinated parameter+ structure 1.0 JSON syntax (already learned well)++Implemented as per-token mask in loss computation:++ loss = sum(CE(logits, targets) * token_weights) / sum(token_weights)++The weighting is computed during data preparation by parsing the+target JSON and labeling each token's role. This focuses gradient+signal on the tokens that matter most for tool-call correctness.++Why not uniform weighting:+ - 60-70% of decoder tokens are JSON structure ({, }, [, ], :, ", ,)+ - The model learns structure quickly (99%+ parse rate)+ - Under uniform weighting, gradient signal is dominated by structure+ tokens — the model keeps "improving" on tokens it already gets right+ - 3x weight on names means 3x gradient signal for the critical tokens+++### 3f. Z-Loss Regularization++Small auxiliary loss on logit magnitude:++ z_loss = 1e-4 * mean(logsumexp(logits, axis=-1)^2)++Prevents logits from growing unboundedly during training. Without this,+logits can reach 100+ in magnitude, causing numerical issues in softmax+and making the model overconfident. Z-loss keeps logits in a reasonable+range (~10-20) without affecting the argmax (relative ordering preserved).+++### 3g. Multi-Objective Training Schedule++All objectives in a single training loop, but at different frequencies:++ Every step:+ - Text generation loss (CE + z-loss)+ - Matryoshka FFN masking (batch split across widths)+ - Gradient clip (global norm 1.0)+ - Muon/Adam update++ Every 100 steps:+ - QAT (fake INT4 quantization)+ - Contrastive loss (CLIP-style, weight 0.1)++ During pruning epoch (steps 33%-67%):+ - Every prune_interval steps: recompute prune mask+ - Apply mask after each gradient update++ End of pruning epoch:+ - Lock mask, report final sparsity++This schedule minimizes overhead: the expensive objectives (QAT,+contrastive) run infrequently, and their signals are low-noise+enough that per-step computation is unnecessary.+++## 4. Experiment Design++### Exp 1: Encoder-Decoder vs Decoder-Only [MUST-HAVE]++Setup:+ - Train encoder-decoder (our architecture) and decoder-only+ (same total params, same d_model, combined layer count) on+ identical data with identical recipe+ - Decoder-only: concatenate [query | tools | answer] as single+ sequence, causal attention throughout+ - Match total FLOPs as closely as possible++Metrics:+ | Architecture | Params | name_f1 | call_f1 | exact_match | value_acc |++ Also report:+ - Inference latency (tok/s) for both+ - Encoder reuse efficiency (for enc-dec: encode once, decode many)++Hypothesis:+ - Enc-dec should win on name_f1 (bidirectional tool encoding)+ - Enc-dec should win on multi-tool examples (better tool discrimination)+ - Dec-only may be competitive on single-tool examples+ - Enc-dec inference faster when generating multiple calls per context+ (encoder output cached, decoder runs multiple times)++What this shows:+ - The architectural choice is justified for tool calling+ - Bidirectional encoding of tool definitions helps discrimination+++### Exp 2: Full Recipe Ablation [MUST-HAVE]++Setup:+ - Full recipe: all components enabled+ - Drop-one ablations (each removes exactly one component):+ (a) No Muon (Adam-only for all params)+ (b) No QAT (no fake quantization)+ (c) No pruning (no magnitude pruning)+ (d) No structured loss weighting (uniform weights)+ (e) No Matryoshka (full FFN width only)+ (f) No contrastive (already in tool-selection.txt Exp 2)+ (g) No z-loss+ (h) No DReLU (use SwiGLU instead)++Metrics:+ | Condition | val_ppl | name_f1 | call_f1 | quant_ppl | sparse_ppl |+ |---------------------|---------|---------|---------|-----------|------------|+ | Full recipe | | | | | |+ | - Muon | | | | | |+ | - QAT | | | | N/A | |+ | - Pruning | | | | | N/A |+ | - Struct. weights | | | | | |+ | - Matryoshka | | | | | |+ | - z-loss | | | | | |+ | - DReLU (+SwiGLU) | | | | | |++Hypothesis:+ - Full recipe is best on aggregate+ - Dropping structured weighting hurts name_f1 most (name tokens+ lose 3x gradient signal)+ - Dropping Muon hurts convergence speed (need more epochs)+ - Dropping QAT: val_ppl similar, but quant_ppl much worse+ - Dropping pruning: val_ppl similar, but sparse_ppl worse+ - Each component contributes — no single one is dominant++What this shows:+ - The recipe is co-designed: each piece has a role+ - No single component carries the result — it's the combination+++### Exp 3: Matryoshka Accuracy-Efficiency Pareto [MUST-HAVE]++Setup:+ - Single trained model with Matryoshka FFN+ - Evaluate at FFN widths: full (2048), 2x (1024), 4x (512), 8x (256)+ - Compare with separately trained models at each width+ - Compare with knowledge distillation (full → small)++Metrics:+ | Method | d_ff | Params | FLOPs | val_ppl | call_f1 |+ |-----------------|------|--------|-------|---------|---------|+ | Matryoshka full | 2048 | | | | |+ | Matryoshka 2x | 1024 | | | | |+ | Matryoshka 4x | 512 | | | | |+ | Matryoshka 8x | 256 | | | | |+ | Separate full | 2048 | | | | |+ | Separate 2x | 1024 | | | | |+ | Separate 4x | 512 | | | | |+ | Separate 8x | 256 | | | | |+ | Distilled 2x | 1024 | | | | |+ | Distilled 4x | 512 | | | | |++ Plot: Pareto curve (FLOPs vs call_f1) for each method++Hypothesis:+ - Matryoshka sub-models are within 1-3% of separately trained models+ - Matryoshka beats distillation (no teacher-student mismatch)+ - Single checkpoint, multiple operating points — simpler deployment++What this shows:+ - Matryoshka is a practical elastic inference solution+ - The accuracy-efficiency tradeoff is smooth (no cliff)+ - One model replaces N separately trained models+++### Exp 4: Optimizer Comparison [STRONG]++Setup:+ - Muon+Adam (our split): Muon for kernels, Adam for rest+ - Adam-only: AdamW with same schedule for all params+ - SGD+momentum: baseline optimizer+ - Same architecture, same data, same total steps++Metrics:+ - Training loss curves (convergence speed)+ - Final val_ppl and call_f1+ - Wall-clock time per step++Hypothesis:+ - Muon+Adam converges fastest (fewer steps to same loss)+ - Adam-only reaches similar final quality but slower+ - SGD is significantly worse (no adaptive rates for heterogeneous params)+ - Muon adds negligible wall-clock overhead (Newton-Schulz is 5 matmuls)++What this shows:+ - Muon accelerates convergence for attention/FFN matrices+ - The split optimizer is justified: different params need different treatment+++### Exp 5: QAT + Pruning Compression Analysis [STRONG]++Setup:+ - Evaluate combinations:+ Full precision, no pruning (baseline)+ INT4 quantized (post-training, no QAT)+ INT4 quantized (with QAT)+ 50% pruned (no QAT)+ INT4 + 50% pruned (both, with QAT)+ - Report model size, inference speed, and accuracy++Metrics:+ | Condition | Size(MB) | tok/s | val_ppl | call_f1 |+ |----------------------|----------|-------|---------|---------|+ | FP16 baseline | | | | |+ | INT4 (no QAT) | | | | |+ | INT4 (with QAT) | | | | |+ | 50% pruned | | | | |+ | INT4 + 50% pruned | | | | |++ Also: group_size sweep for QAT {16, 32, 64, 128}++Hypothesis:+ - QAT recovers most of the INT4 quantization loss (quant_ppl gap < 0.5)+ - Without QAT, INT4 has significant degradation at small model sizes+ - Pruning + QAT compound: 8x size reduction with <5% accuracy loss+ - Group size 32 is sweet spot (matches hardware vector width)++What this shows:+ - QAT and pruning are essential for edge deployment+ - Co-designed training (both at group_size=32) enables extreme compression+ - Small models benefit more from QAT than large models (fewer redundant params)+++### Exp 6: Activation Comparison (DReLU vs SwiGLU vs GeGLU) [STRONG]++Setup:+ - Three activations, everything else identical+ - Report: accuracy, activation sparsity, inference speed++Metrics:+ | Activation | val_ppl | call_f1 | sparsity% | tok/s | quant_ppl |++Hypothesis:+ - DReLU has highest activation sparsity (~65%)+ - SwiGLU has best raw val_ppl (smoothest gradients)+ - DReLU wins on call_f1 (sharper features for structured output)+ - DReLU wins on tok/s (no sigmoid/erf, sparse activations)+ - DReLU has best quant_ppl (sparse activations → better quantization)++What this shows:+ - DReLU is the right choice for tool calling + edge deployment+ - The sparsity-accuracy tradeoff favors DReLU in this domain+++### Exp 7: Structured Loss Weight Sensitivity [STRONG]++Setup:+ - Vary name weight: {1.0, 2.0, 3.0, 5.0, 10.0}+ - Vary value weight: {1.0, 2.0, 3.0, 5.0}+ - Key weight fixed at 1.5x++Metrics (heatmap: name_weight x value_weight):+ - name_f1+ - call_f1+ - value_acc++Hypothesis:+ - name_weight=1 (uniform): worst name_f1+ - name_weight=3: sweet spot (sufficient signal without overwhelming)+ - name_weight=10: name_f1 saturates, value_acc degrades (gradient+ starved on value tokens)+ - value_weight=2: good balance++What this shows:+ - Structured weighting matters for structured output+ - The weights should reflect the error impact hierarchy:+ wrong name (catastrophic) > wrong value (incorrect) > wrong key (hallucinated)+++### Exp 8: Training Curriculum Effect [NICE-TO-HAVE]++Setup:+ - With curriculum: sort by tool count (easy → hard within epoch)+ - Without curriculum: random shuffle+ - Same total data, same epochs++Metrics:+ - val_ppl convergence curve+ - Per-tool-count call_f1 at epoch 1, 5, 10++Hypothesis:+ - Curriculum helps early training (faster initial convergence)+ - By end of training, gap narrows+ - Biggest benefit on high-tool-count examples (model learns+ simple patterns first, then extends to complex cases)++What this shows:+ - Curriculum is useful but not critical+ - Justifies the easy→hard ordering for tool-call training+++### Exp 9: ZCRMSNorm vs Standard RMSNorm [NICE-TO-HAVE]++Setup:+ - ZCRMSNorm (γ init to 0, applied as (1+γ)*x/RMS)+ - Standard RMSNorm (γ init to 1, applied as γ*x/RMS)+ - Same architecture, same recipe++Metrics:+ - Training loss curves (first 1000 steps: stability)+ - Final val_ppl and call_f1+ - Gradient norm trajectory++Hypothesis:+ - ZCRMSNorm has smoother early training (no initial activation spikes)+ - Both converge to similar final quality+ - ZCRMSNorm shows lower gradient norm variance in first 500 steps++What this shows:+ - Zero-centering helps training stability, especially at small scale+ - Free improvement: no added cost, just different initialization+++### Priority++Must-have (core claims):+ 1. Enc-dec vs dec-only (Exp 1) — justifies architecture choice+ 2. Full recipe ablation (Exp 2) — proves co-design+ 3. Matryoshka Pareto (Exp 3) — proves elastic inference++Strong additions:+ 4. Optimizer comparison (Exp 4) — justifies Muon+Adam+ 5. QAT + pruning compression (Exp 5) — deployment story+ 6. Activation comparison (Exp 6) — justifies DReLU+ 7. Loss weight sensitivity (Exp 7) — justifies structured weighting++Nice-to-have:+ 8. Curriculum effect (Exp 8) — training efficiency+ 9. ZCRMSNorm ablation (Exp 9) — initialization choicediff --git a/notes/sparsification.txt b/notes/sparsification.txtnew file mode 100644index 0000000..dbf6ef4--- /dev/null+++ b/notes/sparsification.txt@@ -0,0 +1,521 @@+Group-Aligned Sparsification for Quantized Edge Deployment+===========================================================++## 1. Problem++Needle targets tiny edge devices (phones, microcontrollers, wearables)+where memory, compute, and battery are constrained. Two standard+compression techniques exist independently:++ (a) Quantization: reduce weight precision (FP16 → INT4), 4x memory savings+ (b) Pruning: zero out weights, skip multiplications, save compute + energy++The problem: these are typically applied independently, with mismatched+granularity. Standard unstructured pruning zeros individual weights,+but quantization operates on groups (32 consecutive weights share one+scale factor). A group with 31 zeros and 1 nonzero still needs its+full scale factor, full memory, and full compute — the sparsity is+invisible to the quantization kernel.+++## 2. Key Insight: Align Pruning Groups with Quantization Groups++If pruning and quantization use the same group structure, an entire+pruned group becomes a contiguous block of zeros that hardware can+skip completely:++ Weight matrix W: (in_features, out_features)++ Group-aligned view (group_size=32):+ ┌────────────────────────────────────────────┐+ │ group 0: [w_0, w_1, ..., w_31] scale_0 │ ← LIVE (has nonzeros)+ │ group 1: [0, 0, ..., 0] scale_1 │ ← DEAD (all zeros)+ │ group 2: [w_64, w_65, ..., w_95] scale_2 │ ← LIVE+ │ group 3: [0, 0, ..., 0] scale_3 │ ← DEAD+ │ ... │+ └────────────────────────────────────────────┘++ Unstructured sparsity (misaligned):+ ┌────────────────────────────────────────────┐+ │ group 0: [w_0, 0, w_2, 0, ..., 0, w_31] │ ← half zeros, but+ │ scale_0 still needed │ can't skip group+ │ all 32 multiplications still run │+ └────────────────────────────────────────────┘++ Group-aligned sparsity:+ ┌────────────────────────────────────────────┐+ │ group 1: [0, 0, 0, 0, ..., 0, 0] │ ← entire group zero+ │ scale_1 can be skipped │ skip 32 MACs+ │ skip memory load for 32 weights │ skip scale load+ │ skip 32 multiplications │ zero energy+ └────────────────────────────────────────────┘++When a full group is zero:+ - Skip loading 32 INT4 weights from memory (16 bytes saved per group)+ - Skip loading the group scale factor (2 bytes saved)+ - Skip 32 multiply-accumulate operations+ - Skip the dequantization step for that group+ - On battery-powered devices: fewer memory accesses = less energy++The key: our QAT and pruning both use group_size=32. This is not+coincidence — they are co-designed to produce hardware-friendly+compressed models.+++## 3. Architecture++Three components: group-wise magnitude scoring, cubic gradual schedule,+and mask application.++### 3a. Group-Wise Magnitude Scoring++For each weight matrix, divide the input dimension into groups of 32:++ W ∈ R^{in × out} → W_grouped ∈ R^{num_groups × 32 × out}++ Score each group by the L1 norm of its elements:++ score_g = Σ_{i=0}^{31} Σ_{j=0}^{out-1} |W_grouped[g, i, j]|++ This gives one score per group. Groups with low total magnitude+ contribute least to the output and are pruned first.++Global threshold: all group scores across ALL weight matrices are+collected and sorted. The threshold is set at the s-th percentile,+where s is the target sparsity. This means pruning is globally+balanced — a layer with many unimportant groups loses more than a+layer with all important groups.++ all_scores = concat([scores from every weight matrix])+ threshold = percentile(all_scores, sparsity * 100)+ mask[g] = 1 if score[g] > threshold else 0++Binary mask: the mask has the same shape as the weight matrix.+All 32 weights in a pruned group are zeroed together.++ mask_grouped[g, :, :] = broadcast(score[g] > threshold)+ mask = mask_grouped.reshape(in_features, out_features)++### 3b. Handling nn.scan Parameters++nn.scan stacks layer parameters as 3D tensors: (num_layers, in, out).+Each layer is pruned independently — different layers can have different+sparsity patterns, but the global threshold is shared.++ For W ∈ R^{num_layers × in × out}:+ for each layer l:+ W[l] → group → score → collect into global pool+ global threshold from all scores across all layers and all matrices+ for each layer l:+ W[l] → group → mask using global threshold++This ensures pruning pressure is distributed based on actual weight+magnitude, not layer position.+++### 3c. Cubic Gradual Schedule++One-shot pruning (remove 50% of groups instantly) causes irrecoverable+accuracy loss. Instead, gradually increase sparsity following a cubic+schedule (Zhu & Gupta 2017):++ s(t) = s_final × (1 - (1 - (t - t_start) / (t_end - t_start))³)++ ┌─────────────────────────────────────────────────┐+ │ sparsity │+ │ ▲ s_final ─── ─── ─── ─── ─── ──│+ │ │ ╱ │+ │ │ ╱ │+ │ │ ╱ ← fast ramp │+ │ │ ╱ │+ │ │ ╱ │+ │ │ _╱ ← slow start │+ │ │ __╱ │+ │ │ ___╱ │+ │ │ ────╱ │+ │ 0%─────┬──────────────────┬─────────────→ step │+ │ t_start (33%) t_end (67%) │+ └─────────────────────────────────────────────────┘++ Default schedule within pruning epoch:+ t_start = 33% of epoch steps (first third: no pruning)+ t_end = 67% of epoch steps (middle third: gradual pruning)+ remaining third: mask locked, fine-tune surviving weights++ Mask recomputation: every prune_interval=100 steps during the+ ramp, recompute the mask at the current sparsity level. Between+ recomputations, the mask is frozen — weights can recover.++Why cubic (not linear):+ - Slow start: remove only the most obviously unimportant groups first.+ The model barely notices.+ - Fast middle: bulk of pruning happens here, while the model has+ already adapted to losing some groups.+ - Slow end: final fine-grained pruning. The remaining groups are+ closer in importance — take care removing the last few.++After pruning epoch completes:+ - Mask is permanently locked+ - All subsequent training respects the mask (w = w * mask after each step)+ - The model fine-tunes only the surviving weights+++### 3d. Mask Application in Training Loop++The mask is applied after each gradient update, not during:++ # In _train_step:+ state = state.apply_gradients(grads=grads) # update ALL weights+ new_params = cond(+ do_prune,+ lambda: tree_map(lambda w, m: w * m, state.params, prune_mask),+ lambda: state.params,+ )+ state = state.replace(params=new_params)++This is "mask-after-update" (not "mask-before-gradient"). Both are+used in practice, but mask-after-update has an advantage: the optimizer+state (momentum, variance estimates) remains valid for pruned weights.+If we ever want to "un-prune" during the gradual schedule (a group+recovers importance), the optimizer can immediately use its momentum —+no cold start.++In practice, during the cubic ramp, groups can be un-pruned: a group+zeroed at 20% sparsity might survive at 25% sparsity if the global+threshold shifts. This is a natural consequence of global thresholding+with recomputation.+++## 4. Co-Design with INT4 Quantization++### 4a. Shared Group Size++Both QAT and pruning use group_size=32. This is the critical co-design:++ Quantization groups:+ W_grouped = W.reshape(num_groups, 32, out_features)+ scale[g] = max(|W_grouped[g]|) / 7 (one scale per group)+ W_q[g] = clip(round(W_grouped[g] / scale[g]), -8, 7)++ Pruning groups:+ W_grouped = W.reshape(num_groups, 32, out_features)+ score[g] = sum(|W_grouped[g]|)+ mask[g] = score[g] > threshold++ Same reshape. Same groups. The pruned groups are exactly the groups+ the quantization kernel operates on.++### 4b. Hardware Execution Model++On edge INT4 kernels (ARM NEON, Apple AMX, RISC-V V):++ For each output element y[j]:+ for each group g in [0, num_groups):+ if mask[g] == 0: ← group is pruned+ continue ← skip entirely: no load, no compute+ load scale[g] ← 2 bytes+ load W_q[g, :, j] ← 16 bytes (32 × 4-bit)+ dequant = W_q * scale ← 32 multiplies+ y[j] += dot(dequant, x[g*32:(g+1)*32]) ← 32 MACs+ end++ With 50% group sparsity:+ - 50% fewer memory loads (weights + scales)+ - 50% fewer multiply-accumulate operations+ - ~50% less energy (memory access dominates energy on edge)+ - Model size: 50% fewer groups × (16 bytes weights + 2 bytes scale)+ = 50% size reduction ON TOP of 4x from INT4++ Combined compression:+ FP16 baseline: 100% (2 bytes per weight)+ INT4 only: 25% (0.5 bytes per weight + scale overhead)+ INT4 + 50% sparse: ~13% (half the INT4 groups)++ For a 50M parameter model:+ FP16: 100 MB+ INT4: 25 MB (+ ~2 MB scales)+ INT4 + 50% sparse: ~14 MB+++### 4c. Why Group Size 32++ - ARM NEON: 128-bit SIMD → 32 INT4 values per register+ - Apple AMX: operates on blocks that are multiples of 32+ - CUDA: warp size 32, INT4 tensor cores operate on 32-element groups+ - RISC-V V: typical VLEN=128+ → 32+ INT4 elements per vector register++Group size 32 means each pruned group corresponds to exactly one+hardware vector operation that can be skipped. Smaller groups (16)+waste half the register; larger groups (64) are too coarse for+fine-grained pruning. 32 is the hardware-natural sweet spot.+++## 5. Comparison with Other Pruning Approaches++### vs. Unstructured Pruning (SparseGPT, Wanda)++ Unstructured: zeros individual weights anywhere in the matrix.+ 50% unstructured sparsity → every group still has ~half its weights.+ No group can be skipped. Requires sparse matmul kernels (CSR/CSC),+ which have poor hardware utilization on edge devices.++ Group-aligned: zeros entire groups. 50% sparsity → 50% of groups+ are completely empty. Dense matmul kernels skip entire groups.+ No sparse format overhead. No index storage.++ Accuracy comparison:+ Unstructured can achieve slightly better accuracy at the same+ sparsity level (finer granularity → more choices). But the+ speedup is often zero on real hardware without sparse kernel support.++ Our trade: slightly coarser pruning for guaranteed hardware speedup.++### vs. N:M Structured Sparsity (2:4 on NVIDIA Ampere+)++ N:M sparsity: exactly N zeros in every M consecutive elements.+ 2:4 sparsity on Ampere/Hopper: hardware-native 2x speedup.++ Limitations:+ - Only available on NVIDIA Ampere+ GPUs (not edge devices)+ - Fixed 50% sparsity (can't do 30% or 70%)+ - Fixed pattern (2:4) — no flexibility+ - Not compatible with INT4 group quantization grouping++ Our approach: works on any hardware with vector instructions.+ Variable sparsity (0-90%). Compatible with INT4 group quantization.+ No special hardware required.++### vs. Channel/Filter Pruning++ Channel pruning: removes entire rows/columns of weight matrices.+ Maximally structured (every output using that channel is affected).++ Limitation: too coarse. Removing an entire channel of d_model=512+ is a huge capacity hit. For small models, even removing one channel+ is noticeable.++ Our group pruning: operates at the granularity of 32 input features+ per output. Much finer than channel pruning. A matrix can lose+ some groups in one region while keeping others — the pruning+ pattern adapts to the weight distribution.+++## 6. Experiment Design++### Exp 1: Group-Aligned vs Unstructured Sparsity [MUST-HAVE]++Setup:+ - Same model, same training, three pruning modes:+ (a) Group-aligned (group_size=32, our approach)+ (b) Unstructured (individual weight magnitude pruning)+ (c) No pruning (baseline)+ - Sparsity levels: 0%, 25%, 50%, 70%, 90%+ - All with QAT (INT4, group_size=32)++Metrics:+ | Sparsity | Mode | val_ppl | call_f1 | quant_ppl | actual_speedup |+ |----------|---------------|---------|---------|-----------|----------------|+ | 0% | baseline | | | | 1.0x |+ | 25% | unstructured | | | | |+ | 25% | group-aligned | | | | |+ | 50% | unstructured | | | | |+ | 50% | group-aligned | | | | |+ | 70% | unstructured | | | | |+ | 70% | group-aligned | | | | |+ | 90% | unstructured | | | | |+ | 90% | group-aligned | | | | |++ actual_speedup: measured on target edge device (ARM CPU with NEON)++Hypothesis:+ - Unstructured has slightly better accuracy at same sparsity (~1-2%)+ - Group-aligned has MUCH better actual speedup (near-linear with sparsity)+ - Unstructured has ~1x speedup (no hardware skip support on edge)+ - Group-aligned at 50% matches unstructured at 50% accuracy but with+ real 1.8-2x speedup+ - The quant_ppl gap between aligned/unstructured is small because QAT+ adapts to both patterns++What this shows:+ - Group alignment is essential for actual deployment speedup+ - The small accuracy cost is worth the massive efficiency gain+ - Unstructured sparsity is a "paper metric" — looks good, no real speedup+++### Exp 2: Gradual vs One-Shot Pruning [MUST-HAVE]++Setup:+ - Same target sparsity (50%), three schedules:+ (a) Gradual cubic (our approach: 0% → 50% over 33%-67% of epoch)+ (b) One-shot at epoch start (prune to 50% immediately, then train)+ (c) One-shot at epoch end (train, then prune to 50%)+ - Same total training budget++Metrics:+ | Schedule | val_ppl | call_f1 | name_f1 | value_acc |++ Also: val_ppl convergence curve (how quickly accuracy recovers)++Hypothesis:+ - Gradual cubic is best (model adapts smoothly)+ - One-shot at start: worst (massive capacity loss before adaptation)+ - One-shot at end: middle (full training, then sudden pruning)+ - Gradual reaches same final quality 30-50% faster than one-shot-start++What this shows:+ - Gradual pruning is necessary for small models+ - The cubic schedule is justified (smooth ramp > abrupt cut)+++### Exp 3: Group Size Sweep [MUST-HAVE]++Setup:+ - Group sizes: 8, 16, 32, 64, 128+ - QAT group size always matches pruning group size+ - 50% target sparsity++Metrics:+ | group_size | val_ppl | call_f1 | quant_ppl | model_size(MB) | tok/s(edge) |+ |------------|---------|---------|-----------|----------------|-------------|+ | 8 | | | | | |+ | 16 | | | | | |+ | 32 | | | | | |+ | 64 | | | | | |+ | 128 | | | | | |++Hypothesis:+ - group_size=8: best accuracy (finest granularity), worst speedup+ (groups don't fill hardware vectors)+ - group_size=32: sweet spot (matches hardware vector width)+ - group_size=128: worst accuracy (too coarse), but fastest per-group+ skip (fewer groups to iterate)+ - Quantization overhead (scale storage) is lowest at group_size=128+ but accuracy suffers++What this shows:+ - group_size=32 is the right choice (hardware-natural + good accuracy)+ - Justifies the co-design: QAT and pruning both at 32+++### Exp 4: Combined Compression Frontier [STRONG]++Setup:+ - Sweep the sparsity-quantization space:+ FP16 + {0%, 25%, 50%, 70%} sparsity+ INT8 + {0%, 25%, 50%, 70%} sparsity+ INT4 + {0%, 25%, 50%, 70%} sparsity+ - All with QAT (matched to quantization level)+ - Report model size, inference speed, and accuracy++Metrics:+ Plot: model_size(MB) vs call_f1 (Pareto frontier)+ Plot: tok/s(edge) vs call_f1 (Pareto frontier)++ Annotate each point with (quant_level, sparsity%)++Hypothesis:+ - INT4 + 50% sparse dominates the Pareto frontier for edge+ - FP16 + 0% is best accuracy but impractical (too large/slow)+ - INT4 + 0% is a strong baseline but leaving performance on table+ - INT4 + 70% pushes into extreme compression with acceptable loss++What this shows:+ - Quantization and sparsification are complementary (multiplicative compression)+ - The co-designed approach (shared group_size) maximizes both+ - Practical deployment recipe: INT4 + 50% sparse ≈ 8x compression+ with <5% accuracy loss+++### Exp 5: Energy Measurement on Edge Device [STRONG]++Setup:+ - Deploy on target edge device (e.g. Raspberry Pi 5, Pixel phone)+ - Measure per-inference energy (joules) using hardware power monitor+ - Compare: FP16 baseline, INT4-only, INT4+50% group-sparse++Metrics:+ | Condition | Energy(mJ) | Latency(ms) | call_f1 | Model size |+ |-------------------|------------|-------------|---------|------------|+ | FP16 baseline | | | | |+ | INT4 | | | | |+ | INT4 + 50% sparse | | | | |++ Also: energy breakdown by component (memory load, compute, other)++Hypothesis:+ - Memory loads dominate energy on edge (>60% of total)+ - INT4 + 50% sparse reduces memory energy by ~87% (4x quant × 2x sparse)+ - Compute energy scales linearly with sparsity (skip zero groups)+ - Total energy reduction: 5-8x vs FP16 baseline++What this shows:+ - Group-aligned sparsity directly maps to energy savings+ - This is the deployment story: same model, 5-8x less energy per inference+ - Battery life argument for edge devices+++### Exp 6: Pruning Schedule Ablation [STRONG]++Setup:+ - Vary pruning schedule parameters:+ start_frac: {0.1, 0.2, 0.33, 0.5}+ end_frac: {0.5, 0.67, 0.8, 0.9}+ interval: {10, 50, 100, 500}+ - 50% target sparsity++Metrics:+ - Final call_f1 for each combination+ - Training efficiency: steps to reach 90% of final accuracy++Hypothesis:+ - start_frac=0.33, end_frac=0.67 (middle third) is near-optimal+ - Too early start (0.1): model hasn't learned enough to identify+ unimportant groups+ - Too late end (0.9): not enough fine-tuning time after pruning+ - Interval=100 is sufficient (mask doesn't need per-step updates)++What this shows:+ - The default schedule is well-chosen+ - Pruning during the "middle" of training is optimal (model has+ learned structure but still has time to adapt)+++### Exp 7: Sparsity Pattern Visualization [NICE-TO-HAVE]++Setup:+ - Visualize the pruning mask at 50% sparsity+ - Heat maps: which groups survive in each layer+ - Compare encoder vs decoder patterns++Presentation:+ - Matrix visualization: rows = input groups, columns = output features+ - Color: white = pruned, black = kept+ - One plot per layer type (encoder self-attn, decoder cross-attn, FFN)++Hypothesis:+ - FFN layers are more prunable than attention projections+ - Encoder FFN: earlier layers keep more groups (basic features)+ - Decoder cross-attention: keeps groups aligned with tool-relevant+ encoder positions+ - Symmetric pattern in attention Q/K but asymmetric in V/O++What this shows:+ - The model learns which groups matter+ - Visual evidence that global thresholding distributes sparsity+ intelligently across layers+++### Priority++Must-have (core claims):+ 1. Group-aligned vs unstructured (Exp 1) — proves hardware speedup+ 2. Gradual vs one-shot (Exp 2) — proves schedule necessity+ 3. Group size sweep (Exp 3) — proves group_size=32 choice++Strong additions:+ 4. Combined compression frontier (Exp 4) — deployment Pareto+ 5. Energy measurement (Exp 5) — real-world edge impact+ 6. Schedule ablation (Exp 6) — hyperparameter justification++Nice-to-have:+ 7. Sparsity pattern visualization (Exp 7) — interpretabilitydiff --git a/notes/tool-selection.txt b/notes/tool-selection.txtnew file mode 100644index 0000000..f00f5dc--- /dev/null+++ b/notes/tool-selection.txt@@ -0,0 +1,322 @@+Contrastive Tool Selection via Shared Encoder+==============================================++## 1. Problem++Tool-calling models receive a set of tool definitions in-context and must+select the right tool(s) for a given query. Two challenges:++ (a) Context window pressure: with 1024 encoder tokens, fitting many+ tool definitions alongside the query is tight. Real APIs can have+ 50-200 endpoints — they simply don't fit.++ (b) Selection from large catalogs: even if tools fit, the model must+ discriminate between semantically similar tools (e.g.,+ search_flights vs search_hotels vs search_restaurants). This is a+ retrieval problem, not a generation problem.++The standard approach is a separate retrieval model (e.g., embedding API)+to pre-filter tools before passing them to the generator. This adds a+second model, a second inference pass, and a mismatch between the+representations used for retrieval and generation.+++## 2. Key Insight: Shared Encoder, Dual Use++Our encoder already processes both queries and tool definitions during+generation (encoder input = [query | <tools> | tools...]). The same+encoder representations that attend to tools during generation can be+repurposed for retrieval by adding a lightweight contrastive head.++Architecture:++ ┌─────────────┐ ┌─────────────┐+ │ "What is │ │ {"name": │+ │ the weather│ │ "get_weather│+ │ in SF?" │ │ "params": │+ │ │ │ {"location"│+ └──────┬──────┘ │ : "string"}│+ │ │ } │+ ▼ └──────┬──────┘+ ┌──────────────┐ │+ │ │ │+ │ Shared │◄──────────┘+ │ Encoder │+ │ (8 layers) │+ │ │+ └──────┬───────┘+ │+ ┌────┴────┐+ │ │+ ▼ ▼+ ┌─────┐ ┌─────────┐+ │Mean │ │Contras- │+ │Pool │ │tive │+ │ │ │Proj │+ └──┬──┘ │(d→128) │+ │ └────┬────┘+ │ │+ ▼ ▼+ (d_model) (128-d, L2-normalized)+ │ │+ ▼ ▼+ Cross- Cosine similarity+ attention for retrieval+ in decoder++The contrastive projection head is a single linear layer (d_model → 128)+followed by L2 normalization. It's trained with CLIP-style symmetric+cross-entropy loss: in each batch, the B queries are paired with B+positive tools, and the B²-B in-batch negatives provide contrastive signal.++This means:+ - No separate retrieval model — same encoder, same parameters+ - Retrieval representations are grounded in the same space used for+ generation — what the encoder learns for cross-attention also helps+ distinguish tools+ - The contrastive head adds <0.01% parameters (one 512×128 matrix)+++## 3. Training Details++CLIP-style loss with learnable temperature:++ loss = 0.5 * (CE(q·t^T/τ, labels) + CE(t·q^T/τ, labels))++ - q, t ∈ R^{B×128}, L2-normalized+ - τ = exp(log_temp), clamped to [1/100, 100]+ - labels = arange(B) (diagonal is positive)+ - In-batch negatives: B-1 negatives per query, free++Training schedule:+ - Contrastive loss weighted at 0.1× text loss+ - Computed every 100 steps (not every step — saves compute, contrastive+ signal is less noisy so doesn't need per-step updates)+ - Uses the same encoder parameters as text generation (shared, not frozen)++Data construction:+ - Each training example has a query, a list of tools, and the answer+ (which tools were called)+ - For contrastive training: extract (query, positive_tool) pairs where+ positive_tool is any tool that was actually called+ - Each query may have multiple positives; one is randomly sampled per batch+ - Individual tools are tokenized separately (max 256 tokens each)+++## 4. Inference Pipeline++Two-stage inference for large catalogs:++ Stage 1 — Retrieval:+ encode_contrastive(query) → q_emb (128-d)+ encode_contrastive(tools) → t_embs (N×128)+ scores = q_emb @ t_embs.T+ top_k = argsort(-scores)[:k]++ Stage 2 — Generation:+ encoder_input = [query | <tools> | top_k_tools]+ decoder generates tool calls with only the selected tools in context++This naturally handles the context window constraint: even with 200+available tools, we retrieve top-5 and fit them easily in 1024 tokens.+The retrieval and generation use the same encoder weights, so what the+retrieval stage selects is precisely what the generation stage can+attend to effectively.++For small catalogs (≤5 tools): skip retrieval, pass all tools directly.+The contrastive head is never used at inference in this case — it was+only useful as a training regularizer (the shared representations+benefit generation even without explicit retrieval).+++## 5. Why This Design++### vs. Separate retrieval model (e.g., embed API + generator)+ - One model instead of two → simpler deployment, less memory+ - Shared representations → retrieval is aligned with generation+ - No train/inference distribution shift between retriever and generator++### vs. Stuffing all tools in context+ - Doesn't scale: 50 tools × ~50 tokens each = 2500 tokens, exceeds+ 1024 encoder context+ - Even if they fit, attention over many irrelevant tools dilutes signal+ - Retrieval pre-filtering gives the generator a clean, focused context++### vs. Fine-tuning a separate contrastive model+ - The encoder is already "seeing" tools during generation training+ - Adding a projection head extracts what it already knows+ - Multi-task learning: contrastive loss acts as regularizer for the+ encoder, improving generation even on in-context tools++### The regularization effect+ Even when retrieval isn't used at inference (small catalogs), the+ contrastive objective during training forces the encoder to build+ discriminative representations of tools. This helps cross-attention+ in the decoder: the encoder outputs for each tool are more distinct,+ making it easier for the decoder to attend to the right tool.+++## 6. Experiment Design++### Exp 1: Retrieval Quality [MUST-HAVE]++Setup:+ - Validation set, each example has query + tools + ground truth calls+ - Encode query and all tools from that example contrastively+ - Rank tools by cosine similarity, check if positive tools are in top-k++Metrics:+ | k | Recall@k |+ |-----|----------|+ | 1 | |+ | 3 | |+ | 5 | |+ | 10 | |++ Also report MRR (mean reciprocal rank of first positive).++Stratify by:+ - Number of tools per example (harder with more tools)+ - Number of positive tools (multi-call examples)++What this shows:+ - The shared encoder produces discriminative tool representations+ - Recall@5 should be very high (>95%) — retrieval is nearly solved+++### Exp 2: Ablation — Generation Quality With vs Without Contrastive [MUST-HAVE]++Setup:+ - Train two models: with contrastive loss (λ=0.1) and without (λ=0)+ - Same architecture, same data, same hyperparameters otherwise+ - Evaluate generation metrics on val set (no retrieval at inference)++Metrics:+ | Condition | name_f1 | call_f1 | exact_match | value_acc |++Hypothesis:+ - Even without using retrieval at inference, the contrastive-trained+ model should have better name_f1 and call_f1+ - The regularization effect: contrastive loss forces discriminative+ encoder representations → better cross-attention → better generation++What this shows:+ - Contrastive learning isn't just for retrieval — it improves generation+ - This is the key insight: one objective helps two tasks+++### Exp 3: Retrieve-then-Generate vs Full Context [MUST-HAVE]++Setup:+ - Create synthetic scenarios with large tool catalogs (20, 50, 100 tools)+ by combining tools from multiple validation examples+ - Three conditions:+ (a) All tools in context (may truncate)+ (b) Top-5 retrieved tools in context+ (c) Oracle top-5 (ground truth positive tools + random negatives)++Metrics:+ | Condition | 20 tools | 50 tools | 100 tools |+ | | name_f1 call_f1| name_f1 call_f1| name_f1 call_f1|+ |----------------|---------------|---------------|---------------|+ | All in context | | | |+ | Retrieved top-5| | | |+ | Oracle top-5 | | | |++Hypothesis:+ - With 20 tools: "all in context" works but retrieved is comparable+ - With 50+ tools: "all in context" degrades (truncation, attention dilution)+ - Retrieved top-5 stays close to oracle → retrieval solves the scaling problem++What this shows:+ - Retrieval is necessary for real-world tool catalogs+ - The gap between retrieved and oracle quantifies retrieval errors+ - The gap between all-context and retrieved quantifies context pressure+++### Exp 4: Contrastive Loss Weight Sweep [STRONG]++Setup:+ - Train models with λ ∈ {0, 0.01, 0.05, 0.1, 0.2, 0.5}+ - Evaluate both retrieval (Recall@k) and generation (call_f1)++Metrics (line plot, x=λ):+ - Recall@5 (retrieval quality)+ - call_f1 (generation quality)++Hypothesis:+ - λ=0: no retrieval capability, generation baseline+ - λ=0.01-0.1: retrieval improves rapidly, generation also improves+ - λ>0.2: retrieval saturates, generation may degrade (contrastive+ objective starts dominating, encoder optimizes for discrimination+ over representation)++What this shows:+ - Sweet spot exists where both tasks benefit+ - Justifies λ=0.1 as default+++### Exp 5: Embedding Space Visualization [STRONG]++Setup:+ - Encode 500 queries and their associated tools+ - Project to 2D via t-SNE or UMAP+ - Color by: tool category / query intent / positive vs negative++Presentation:+ - Scatter plot showing query-tool alignment+ - Highlight clusters where similar tools group together+ - Show queries landing near their positive tools++What this shows:+ - Visual evidence that the shared space is semantically meaningful+ - Queries and their positive tools are close; negatives are far+ - Tool clusters correspond to functional categories+++### Exp 6: Retrieval Latency [STRONG]++Setup:+ - Measure time to encode and retrieve from catalogs of varying size+ - Compare: our model's encode_contrastive vs. a separate embedding API++Metrics:+ | Catalog size | Encode time | Retrieval time | Total |++What this shows:+ - Retrieval adds negligible latency (single encoder pass + matmul)+ - Amortization: tool embeddings can be precomputed and cached+ - No network round-trip to a separate embedding service+++### Exp 7: Contrastive Frequency Ablation [NICE-TO-HAVE]++Setup:+ - Vary how often contrastive loss is computed: every step, every 10,+ every 100 (current), every 500+ - Same total training budget++Metrics:+ - Recall@5 convergence curve+ - Final call_f1++What this shows:+ - Contrastive signal is low-noise, doesn't need per-step updates+ - Every-100 is sufficient (saves ~50% of contrastive compute)+ - Justifies the design choice+++### Priority++Must-have (core claims):+ 1. Retrieval quality (Exp 1) — proves it works+ 2. Ablation: with/without contrastive (Exp 2) — proves dual benefit+ 3. Retrieve-then-generate scaling (Exp 3) — proves necessity++Strong additions:+ 4. Loss weight sweep (Exp 4) — justifies hyperparameter+ 5. Embedding visualization (Exp 5) — intuition and visual appeal+ 6. Retrieval latency (Exp 6) — practical deployment story++Nice-to-have:+ 7. Contrastive frequency ablation (Exp 7) — training efficiencydiff --git a/notes/tries.txt b/notes/tries.txtnew file mode 100644index 0000000..7dfed76--- /dev/null+++ b/notes/tries.txt@@ -0,0 +1,267 @@+Selective Trie-Constrained Decoding for Tool-Call Generation+=============================================================++## 1. Problem++The model generates JSON tool calls autoregressively. While it learns+JSON structure well (99-100% parse rate), it can still hallucinate+invalid function names or argument keys. The valid tool definitions are+already in the encoder input — we can constrain the decoder to only+produce valid names/keys at inference time.+++## 2. Key Insight: Constrain What's Broken, Not What Works++The remaining errors are semantic, not syntactic:++ Error type Constrainable? Frequency+ ───────────────────────── ────────────── ─────────+ Wrong/hallucinated name YES (trie) Common+ Hallucinated param key YES (trie) Common+ Wrong argument values NO (open-ended) Common+ Missing arguments NO Moderate+ Malformed JSON NO (already 99%) Rare++Full grammar approaches (Outlines, llama.cpp GBNF, guidance) enforce+valid JSON on every token. This adds 10-30% overhead to solve a problem+that doesn't exist. Our approach only constrains the ~5% of tokens that+are inside name/key spans.+++## 3. Architecture++Three components: a character-level trie, a JSON state machine, and+a token masking layer.++### 3a. Trie++Given tools: get_weather(location, unit), get_stock(symbol)++Name trie: Param trie (get_weather):++ (root) (root)+ | / \+ g l u+ | | |+ e o n+ | | |+ t c i+ | | |+ _ a t*+ / \ |+ w s t+ | | |+ e t i+ | | |+ a o o+ | | |+ t c n*+ | |+ h k* (* = terminal node)+ |+ e+ |+ r*++One name trie shared across all tools. One param trie per function,+built from its parameter keys.++### 3b. JSON State Machine++Tracks position in output to know when to constrain:++ ┌──────┐ "name":" ┌─────────┐ closing " ┌──────┐+ │ FREE │────────────→│ IN_NAME │────────────→│ FREE │+ └──────┘ └─────────┘ └──────┘+ │ │+ │ "arguments":{ then {" or ," │+ │ ┌────────────┐ closing " │+ └───────────────→│ IN_ARG_KEY │─────────────→┘+ └────────────┘++State transitions driven by suffix matching on generated text.+Tracks nesting depth to avoid false triggers inside nested values.+Tracks string values with escape handling to avoid false triggers+inside string content.++### 3c. Token Masking++At each constrained step:++ 1. Look up current trie node (root + constrained_buf prefix)+ 2. Gather valid first chars from trie children (+ closing " if terminal)+ 3. For each candidate token (via TokenIndex first-char lookup):+ - Walk trie char-by-char through token text+ - If token contains ", check trie node is terminal at that point+ - Mark as valid or invalid+ 4. Set invalid token logits to -inf+ 5. Fallback: if ALL tokens masked, revert to unconstrained + log warning++Example (IN_NAME state, after consuming "get_"):++ Trie node at "get_" has children: {w, s}+ Scan TokenIndex for tokens starting with 'w' or 's':+ "weather" → walk trie: w→e→a→t→h→e→r, all valid → ALLOW+ "we" → walk trie: w→e, valid prefix → ALLOW+ "stock" → walk trie: s→t→o→c→k, terminal → ALLOW+ "sun" → walk trie: s→u, 'u' not child of 's' → BLOCK++Multi-char token boundary handling:+ Token "ther\"" (closing quote mid-token):+ Walk trie for 't','h','e','r' → valid path+ Hit '"' → check if node is terminal → yes (weather*) → ALLOW+ Correctly handles BPE tokens that span the name boundary.+++## 4. Feedback Effect++The corrected token feeds back into the decoder buffer. All subsequent+generation conditions on the valid name:++ Without constraint: ..."name":"get_weathr","arguments":{"locaton":...+ ~~~~~~~~~~~~ wrong ~~~~~ ~~~~~~~~ wrong ~~~~++ With constraint: ..."name":"get_weather","arguments":{"location":...+ ~~~~~~~~~~~~ forced ~~~~ ~~~~~~~~~ model sees+ correct name, generates+ better arguments++This is why in-place correction beats post-processing — the decoder+sees correct context when generating arguments.+++## 5. Comparison with Full Grammar Approaches++ ┌─────────────────────────────────────────────────────────┐+ │ Full grammar (CFG/PDA) │+ │ │+ │ Constrains EVERY token: brackets, colons, commas, │+ │ string delimiters, value types, nesting, whitespace. │+ │ Guarantees syntactically valid JSON always. │+ │ │+ │ Cost: FSM/PDA transition on every token, large state │+ │ space, precomputed token masks per state, high memory. │+ │ Typical overhead: 10-30% slowdown on all tokens. │+ └─────────────────────────────────────────────────────────┘++ ┌─────────────────────────────────────────────────────────┐+ │ Selective constraint (trie + state machine) │+ │ │+ │ Only constrains tokens inside "name":"___" and │+ │ argument key "___" spans. Everything else │+ │ (structure, values, punctuation) is unconstrained. │+ │ │+ │ Cost: lightweight state machine tracks JSON position │+ │ via suffix matching. Trie walk + token masking only │+ │ activates during name/key spans (~5% of tokens). │+ │ Zero overhead on remaining ~95% of tokens. │+ └─────────────────────────────────────────────────────────┘++Why full grammar is overkill here:+ - 99-100% JSON parse rate means structure is already learned+ - Full grammar pays per-token cost for ~0% parse improvement+ - Can force the model into corners where the most likely valid+ continuation is nonsensical (distorts value generation)++Why selective works:+ - Targets the actual failure mode (hallucinated names/keys)+ - Near-zero cost (only active on ~5% of tokens)+ - Never distorts value generation+ - Corrected tokens improve downstream generation via feedback+++## 6. Implementation Details++ token_strings / TokenIndex: cached per tokenizer (built once)+ Device-to-host transfer: only when state machine is active+ (IN_NAME or IN_ARG_KEY); FREE state stays on jnp.argmax+ Enabled by default; disable with --no-constrained+++## 7. Experiment Design++All trie experiments use our val set (5000 samples) — clean controlled+variable where only the decoding mode changes. Exact reference calls+give precise name_f1, param_halluc, value_acc. Per-tool-count breakdown+already implemented. Reproducible with a seed.++### Exp 1: Ablation — Constrained vs Unconstrained [MUST-HAVE]++Same checkpoint, same val set, three conditions:+ (a) Unconstrained (--no-constrained)+ (b) Name-only constraint (trie on function names, keys unconstrained)+ (c) Full selective constraint (trie on names + argument keys)++ | Condition | name_f1 | call_f1 | exact_match | param_halluc | param_miss | value_acc | json_parse |++Expects: name_f1 improves (a→b), param_halluc drops (b→c), json_parse+unchanged across all three (confirms structure isn't the problem).+Needs a "names_only" mode flag in ConstrainedDecoder.++### Exp 2: Latency Overhead [MUST-HAVE]++500 samples, wall-clock time and tok/s, three conditions, 3 runs each.++ | Condition | tok/s | Overhead |++Also report: % tokens where constraint was active, avg time in+apply_constraints per constrained token.+Expects: near-zero overhead because constraints activate on ~5% of tokens.++### Exp 3: Error Analysis — What Constraints Fix [MUST-HAVE]++Run constrained and unconstrained on same 1000 samples. Categorize:+ (a) Both correct → constraint irrelevant+ (b) Unconstrained wrong, constrained correct → constraint helped+ (c) Both wrong (same error) → constraint irrelevant+ (d) Both wrong (different error) → constraint changed but didn't fix+ (e) Unconstrained correct, constrained wrong → constraint hurt (should be ~0)++For category (b), sub-classify:+ - Misspelled function name (get_weathr → get_weather)+ - Hallucinated function name (fetch_weather → get_weather)+ - Hallucinated parameter key (city → location)+ - Misspelled parameter key (locaton → location)++Present as sankey diagram + table of top-10 fixes with before/after.+Category (e) should be empty — proves constraints are safe.++### Exp 4: Breakdown by Tool Count [STRONG]++Use existing per_count breakdown (1-10 tools per example).+Grouped bar chart: name_f1 and param_halluc per bucket, constrained vs not.++Expects: more tools → more names to confuse → bigger constraint benefit.+Shows constraint value scales with schema complexity.++### Exp 5: Downstream Effect — Name Correction Improves Arguments [STRONG]++Focus on examples where unconstrained gets the name wrong.+Compare value_acc and args_acc when name is wrong vs corrected.++Expects: corrected name → better arguments (feedback effect).+Justifies in-place correction over post-processing.++### Exp 6: Benefit Across Training Epochs [STRONG]++Checkpoints at epochs 1, 3, 5, 7, 10. Each evaluated with and without.+Line plot: call_f1 constrained vs unconstrained across epochs.++Expects: biggest delta mid-training ("almost right" phase), positive+everywhere. Shows constraints are complementary to training.++### Exp 7: Full Grammar Comparison [NICE-TO-HAVE]++Implement Outlines-style full JSON grammar constraint as baseline.+Three-way comparison: unconstrained, full grammar, selective trie.++Expects: full grammar has ~same call_f1 but much higher overhead.+Strongest competitive claim but most implementation work.++### Exp 8: Out-of-Distribution Tool Schemas [NICE-TO-HAVE]++Synthetic tools with names/keys not in training data (20-50 examples).+Constrained vs unconstrained.++Expects: OOD names have highest hallucination rate → biggest delta.+Shows value for custom/long-tail APIs. Needs manual data creation.