Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
No FFN by default
HenryNdubuakubc3a3ddf3767
diff --git a/notes/architecture.txt b/notes/architecture.txtindex c421403..a8e4a91 100644--- a/notes/architecture.txt+++ b/notes/architecture.txt@@ -10,711 +10,48 @@ structured weighting + QAT + pruning) that produces a small modelcompetitive with much larger decoder-only models on tool calling.The architecture choices are interdependent:- - Encoder-decoder enables shared-encoder retrieval (see tool-selection.txt)- - Matryoshka FFN enables elastic deployment at multiple compute budgets- - Structured loss weighting targets the actual error distribution- - QAT + pruning enable deployment on edge devices- - Constrained decoding exploits known schemas (see tries.txt)--Each piece alone is known. The system is the contribution.---## 2. Architecture--### 2a. Why Encoder-Decoder (Not Decoder-Only)--Almost all tool-calling models are decoder-only. We use encoder-decoder-because tool calling has a natural asymmetry:-- Input (encode once): Output (generate):- ┌──────────────────────┐ ┌──────────────────────┐- │ Query: "What's the │ │[{"name":"get_weather"│- │ weather in SF?" │ │ "arguments": │- │ <tools> │ │ {"location":"SF"}} │- │ [{"name":"get_weather│ │ ] │- │ "parameters":{...}}│ └──────────────────────┘- │ {"name":"get_stock" │- │ "parameters":{...}}│- │ ] │- └──────────────────────┘-- Encoder input = context (query + tool definitions) — processed once- Decoder output = structured generation (JSON tool calls) — autoregressive--Benefits:- (a) The encoder processes all tools in parallel via bidirectional- self-attention. A decoder-only model processes tools left-to-right,- which means later tools attend to earlier tools but not vice versa.- For discriminating between semantically similar tools (search_flights- vs search_hotels), bidirectional context helps.-- (b) Encoder output is reused across all decoder steps via cross-attention.- In a decoder-only model, KV cache stores the full context, but the- representation is computed left-to-right with causal masking — no- opportunity for the context to self-refine bidirectionally.-- (c) The encoder can serve double duty: cross-attention for generation- AND contrastive retrieval for tool selection (see tool-selection.txt).- A decoder-only model would need a separate retrieval module.-- (d) Smaller parameter budget. With the same d_model and layer count,- an encoder-decoder can encode context more efficiently because- the encoder layers don't need causal masking, cross-attention- projections, or autoregressive logic.---### 2b. Zero-Centered RMSNorm (ZCRMSNorm)--Standard RMSNorm: γ * x / RMS(x), with γ initialized to 1.-Our ZCRMSNorm: (1 + γ) * x / RMS(x), with γ initialized to 0.-- class ZCRMSNorm(nn.Module):- def __call__(self, x):- scale = self.param("scale", zeros, (x.shape[-1],))- rms = sqrt(mean(x^2, axis=-1, keepdims=True) + eps)- return (1 + scale) * x / rms--At initialization, (1 + 0) * x / RMS(x) = x / RMS(x) — the layer-normalizes but applies no learned scaling. This is similar in spirit-to ReZero (Bachlechner et al. 2020) and fixup initialization: at-init, each sub-layer's contribution is minimal, and the residual-stream dominates. The network starts as a near-identity function-and gradually learns to use each layer.--Why this matters for small models:- - Small models are more sensitive to initialization. Standard RMSNorm- with γ=1 can cause early training instability because pre-norm- layers immediately rescale activations.- - Zero-init means gradients flow cleanly through the residual stream- at the start of training, avoiding the "early chaos" period.--Applied everywhere: encoder pre-norm, decoder pre-norm, QK-norm,-ConvModule norm. Consistent zero-centering throughout.---### 2c. Grouped Query Attention (GQA) with QK-Norm--Standard multi-head attention with grouped KV heads:-- Q: (B, num_heads, T, head_dim) — full heads- K: (B, num_kv_heads, T, head_dim) — grouped (repeated for attention)- V: (B, num_kv_heads, T, head_dim) — grouped-- num_heads=8, num_kv_heads=2 → 4:1 ratio, 75% KV parameter savings--QK-norm: apply ZCRMSNorm to Q and K after projection, before attention.-- q = ZCRMSNorm(q_proj(x))- k = ZCRMSNorm(k_proj(x))- attn = softmax(q @ k^T / sqrt(d_head))--QK-norm prevents attention logit growth as training progresses (the-"logit drift" problem observed in large-scale training). With zero-init-normalization, Q and K start with unit-scale dot products and gradually-learn scale — stable from step 1.--This combination (GQA + QK-norm + ZCRMSNorm) is individually known-(Gemma 2, ViT-22B use QK-norm; GQA from Ainslie et al. 2023) but the-zero-centered variant on Q/K is our specific choice.---### 2d. DReLU Activation--Three activation options: SwiGLU, GeGLU, DReLU. Default is DReLU.-- SwiGLU: silu(gate) * up — standard, smooth- GeGLU: gelu(gate) * up — smoother, slightly more expensive- DReLU: relu(gate) * relu(up) — simplest, sparsest-- h = relu(W_gate @ x) * relu(W_up @ x)- out = W_down @ h--DReLU applies ReLU to both gate and up projections independently.-This produces the sparsest activations: any input where either gate-OR up projection is negative gets zeroed. Empirically, ~60-70% of-hidden units are zero at any given step.--Why DReLU for tool calling:- - Activation sparsity enables hardware-level speedups (skip zero- multiplications). Critical for edge deployment.- - Works synergistically with magnitude pruning: sparse activations- mean many weights have low magnitude → cleaner pruning signal.- - Cheaper than SwiGLU/GeGLU: no sigmoid/erf computation.- - For structured output (JSON), the model doesn't need smooth- gradients for nuanced language — it needs sharp, discriminative- features for selecting the right tool name and argument keys.---### 2e. Matryoshka FFN--Train a single model that contains multiple sub-models at different-FFN widths. At inference, choose the width based on compute budget.-- Full model: d_ff = 2048 (all neurons active)- 2x sub-model: d_ff = 1024 (first 1024 neurons)- 4x sub-model: d_ff = 512 (first 512 neurons)--Implementation: prefix masking on FFN hidden dimension.-- def __call__(self, x, ffn_mask=None):- gate = gate_proj(x) # (B, T, d_ff)- up = up_proj(x) # (B, T, d_ff)- h = relu(gate) * relu(up)- if ffn_mask is not None:- h = h * ffn_mask # zero out neurons beyond width- return down_proj(h)--During training, each batch is split across widths:-- Batch (B=32):- items 0-10: full width (d_ff=2048) — all neurons- items 11-21: 2x width (d_ff=1024) — first half- items 22-31: 4x width (d_ff=512) — first quarter-- All items share the same forward pass (single jit-compiled function).- The ffn_mask varies per item. No separate models, no distillation.--The prefix constraint is key: sub-models always use the FIRST k neurons.-This means the first 512 neurons must be useful on their own, the first-1024 must be useful, etc. The training objective forces the model to-organize information by importance along the FFN dimension — most-important features in early neurons.--At inference:- - Edge device: use 4x sub-model (512 neurons, ~25% FFN compute)- - Server: use full model (2048 neurons, full quality)- - Same checkpoint, same parameters, different masks--Why not distillation:- - Distillation requires training N separate models- - Matryoshka trains one model, gets N sub-models- - Sub-models share attention weights — only FFN width changes- - Parameter estimation for any sub-model is straightforward---### 2f. Shared Encoder with Contrastive Head--See tool-selection.txt for full details. Summary:-- Encoder → mean pool → Dense(d_model, 128) → L2 normalize → cosine similarity-- Same encoder serves:- (a) Cross-attention in decoder (generation)- (b) Contrastive retrieval (tool selection from large catalogs)-- Adds <0.01% parameters (one 512x128 matrix + learnable temperature).---### 2g. Conformer Convolution Module (Optional)--When conv_kernel_size > 0, each encoder block adds a convolution module-between self-attention and FFN:-- self-attn → ConvModule → FFN-- ConvModule:- pointwise up (d → 2d) → GLU gate → depthwise conv (k=31) →- ZCRMSNorm → pointwise down (d → d)--This is the Conformer (Gulati et al. 2020) pattern. The depthwise-convolution captures local patterns (token n-grams in tool definitions,-local acoustic patterns in speech). Complementary to global self-attention.--Primary motivation: speech pathway benefits significantly from local-convolution. For text-only, conv_kernel_size=0 (disabled).---### 2h. Speech Pathway--Optional speech input via FastConformer-style 8x subsampling:-- Input: (B, T, 80) mel spectrogram- Output: (B, T//8, d_model) features fed to shared encoder-- Subsampling (3 stages of stride-2 = 8x total):- Conv2D(1→C, k=3, s=2) + ReLU- DW-Conv(C, k=3, s=2) + PW-Conv(C→C) + ReLU- DW-Conv(C, k=3, s=2) + PW-Conv(C→C) + ReLU- Reshape + Linear → d_model-- SpecAugment (training only):- 2 time masks (max 100 frames)- 1 frequency mask (max 27 bins)--The speech subsystem converts audio to the same representation space-as text. After subsampling, the encoder processes speech features-identically to text embeddings — same attention, same FFN, same-contrastive head. Unified encoder for both modalities.---### 2i. Memory-Efficient Layer Stacking--Both encoder and decoder use nn.scan with remat:-- ScanBlock = nn.scan(- nn.remat(BlockBody),- variable_axes={"params": 0}, # stack params along axis 0- split_rngs={"params": True},- length=num_layers,- )-- - nn.scan: shares the block's Python code across layers, stacking- parameters as (num_layers, ...) tensors. Reduces compilation time.- - nn.remat: recomputes activations during backward pass instead of- storing them. Trades ~30% more compute for ~60% less memory.--For a small model this matters less, but it enables scaling experiments-with deeper models on the same hardware.---## 3. Training Recipe--### 3a. Muon + Adam Split Optimizer--Not all parameters benefit from the same optimizer. We split:-- Muon optimizer: all Dense kernels (2D+ tensors)- - Orthogonalize gradients via Newton-Schulz polar decomposition- - Nesterov momentum (μ=0.95)- - Weight decay 0.01- - Separate WSD learning rate schedule-- Adam optimizer: everything else (embeddings, norms, biases, 1D params)- - AdamW with β2=0.95 (lower than default 0.999 for stability)- - No weight decay- - Separate WSD learning rate schedule--Muon's gradient orthogonalization:-- def _newton_schulz(G, steps=5):- a, b, c = 3.4445, -4.7750, 2.0315- X = G / norm(G)- for _ in range(steps):- A = X @ X^T- B = b*A + c*A@A- X = a*X + B@X- return X-- This approximates the polar decomposition G = U*S*V^T and returns U*V^T.- The gradient is projected onto the nearest orthogonal matrix.--Why Muon for attention/FFN kernels:- - Orthogonalized updates prevent weight matrix collapse (rank- deficiency) in small models where d_model is limited.- - Newton-Schulz is cheap: 5 iterations of matrix multiply, no SVD.- - Empirically, Muon converges faster than Adam on projection matrices.--Why Adam for the rest:- - Embeddings are lookup tables — orthogonalization doesn't apply.- - Norm scales are 1D — orthogonalization is meaningless.- - Adam's adaptive learning rates are better for heterogeneous params.---### 3b. Warmup-Stable-Decay (WSD) Schedule-- ┌─────────────────────────────────────────────────────────┐- │ LR │- │ ▲ │- │ │ ┌──────────────────────────────┐ │- │ │ /│ stable │\ │- │ │ / │ │ \ cosine │- │ │ / │ │ \ decay │- │ │ / │ │ \ │- │ │ / │ │ \____ │- │ │/ warmup │ │- │ └──────┴──────────────────────────────┴────────→ step │- │ 0% ~5% 85% 100% │- └─────────────────────────────────────────────────────────┘-- - Warmup: linear ramp from 0 to peak (first ~5% of steps)- - Stable: hold at peak (majority of training)- - Decay: cosine anneal to 5% of peak (last 15% of steps)--Applied separately to Adam and Muon with different peak values.-WSD is simpler than cosine-with-restarts and matches or beats it-for fixed-budget training (MiniCPM, Zhai et al. 2022).---### 3c. Quantization-Aware Training (QAT)--Every 100 steps, apply fake INT4 quantization to all Dense kernels:-- def _fake_quantize_int4(w, group_size=32):- # Group along input dimension- w_grouped = w.reshape(num_groups, group_size, out_feat)- # Per-group symmetric quantization- scale = max(abs(w_grouped), axis=1) / 7.0- w_q = clip(round(w_grouped / scale), -8, 7) * scale- # Straight-through estimator- return w + stop_gradient(w_q - w)-- - Symmetric INT4: 4-bit integers [-8, 7] with per-group scale- - Group size 32: each 32 consecutive input features share a scale- - STE: forward uses quantized weights, backward uses real gradients- - Applied conditionally: only every 100 steps (reduces overhead)--The model learns to be robust to INT4 quantization during training.-At deployment, weights are actually quantized to INT4 → 4x memory-reduction, hardware-accelerated INT4 matmul on supporting devices.--Why every 100 steps instead of every step:- - QAT adds ~15% overhead per step (extra quantize + forward pass)- - Every 100 steps: <0.15% amortized overhead- - Weights change slowly — quantization noise from step N is still- representative at step N+99- - Same rationale as contrastive loss frequency (see tool-selection.txt)---### 3d. Group-Aligned Gradual Pruning--See sparsification.txt for full details. Summary:-- - Group-wise magnitude pruning with group_size=32 — same groups as QAT- - Cubic gradual schedule: 0% → target over middle third of epoch- - Global threshold across all layers and matrices- - Mask applied after gradient update, locked after pruning epoch-- The critical co-design: pruned groups are exactly the groups the INT4- quantization kernel operates on. A pruned group = zero weights + zero- scale factor = zero memory loads + zero MACs. On edge INT4 kernels,- 50% group sparsity → 50% fewer memory accesses and 50% fewer compute- operations, directly translating to battery savings.---### 3e. Structured Per-Token Loss Weighting--Not all tokens in the decoder output are equally important:-- [{"name":"get_weather","arguments":{"location":"San Francisco"}}]-- Token role Weight Rationale- ────────── ────── ─────────- name value 3.0 Wrong name = completely wrong call- arg value 2.0 Wrong value = wrong behavior- arg key 1.5 Wrong key = hallucinated parameter- structure 1.0 JSON syntax (already learned well)--Implemented as per-token mask in loss computation:-- loss = sum(CE(logits, targets) * token_weights) / sum(token_weights)--The weighting is computed during data preparation by parsing the-target JSON and labeling each token's role. This focuses gradient-signal on the tokens that matter most for tool-call correctness.--Why not uniform weighting:- - 60-70% of decoder tokens are JSON structure ({, }, [, ], :, ", ,)- - The model learns structure quickly (99%+ parse rate)- - Under uniform weighting, gradient signal is dominated by structure- tokens — the model keeps "improving" on tokens it already gets right- - 3x weight on names means 3x gradient signal for the critical tokens---### 3f. Z-Loss Regularization--Small auxiliary loss on logit magnitude:-- z_loss = 1e-4 * mean(logsumexp(logits, axis=-1)^2)--Prevents logits from growing unboundedly during training. Without this,-logits can reach 100+ in magnitude, causing numerical issues in softmax-and making the model overconfident. Z-loss keeps logits in a reasonable-range (~10-20) without affecting the argmax (relative ordering preserved).---### 3g. Multi-Objective Training Schedule--All objectives in a single training loop, but at different frequencies:-- Every step:- - Text generation loss (CE + z-loss)- - Matryoshka FFN masking (batch split across widths)- - Gradient clip (global norm 1.0)- - Muon/Adam update-- Every 100 steps:- - QAT (fake INT4 quantization)- - Contrastive loss (CLIP-style, weight 0.1)-- During pruning epoch (steps 33%-67%):- - Every prune_interval steps: recompute prune mask- - Apply mask after each gradient update-- End of pruning epoch:- - Lock mask, report final sparsity--This schedule minimizes overhead: the expensive objectives (QAT,-contrastive) run infrequently, and their signals are low-noise-enough that per-step computation is unnecessary.---## 4. Experiment Design--### Exp 1: Encoder-Decoder vs Decoder-Only [MUST-HAVE]--Setup:- - Train encoder-decoder (our architecture) and decoder-only- (same total params, same d_model, combined layer count) on- identical data with identical recipe- - Decoder-only: concatenate [query | tools | answer] as single- sequence, causal attention throughout- - Match total FLOPs as closely as possible--Metrics:- | Architecture | Params | name_f1 | call_f1 | exact_match | value_acc |-- Also report:- - Inference latency (tok/s) for both- - Encoder reuse efficiency (for enc-dec: encode once, decode many)--Hypothesis:- - Enc-dec should win on name_f1 (bidirectional tool encoding)- - Enc-dec should win on multi-tool examples (better tool discrimination)- - Dec-only may be competitive on single-tool examples- - Enc-dec inference faster when generating multiple calls per context- (encoder output cached, decoder runs multiple times)--What this shows:- - The architectural choice is justified for tool calling- - Bidirectional encoding of tool definitions helps discrimination---### Exp 2: Full Recipe Ablation [MUST-HAVE]--Setup:- - Full recipe: all components enabled- - Drop-one ablations (each removes exactly one component):- (a) No Muon (Adam-only for all params)- (b) No QAT (no fake quantization)- (c) No pruning (no magnitude pruning)- (d) No structured loss weighting (uniform weights)- (e) No Matryoshka (full FFN width only)- (f) No contrastive (already in tool-selection.txt Exp 2)- (g) No z-loss- (h) No DReLU (use SwiGLU instead)--Metrics:- | Condition | val_ppl | name_f1 | call_f1 | quant_ppl | sparse_ppl |- |---------------------|---------|---------|---------|-----------|------------|- | Full recipe | | | | | |- | - Muon | | | | | |- | - QAT | | | | N/A | |- | - Pruning | | | | | N/A |- | - Struct. weights | | | | | |- | - Matryoshka | | | | | |- | - z-loss | | | | | |- | - DReLU (+SwiGLU) | | | | | |--Hypothesis:- - Full recipe is best on aggregate- - Dropping structured weighting hurts name_f1 most (name tokens- lose 3x gradient signal)- - Dropping Muon hurts convergence speed (need more epochs)- - Dropping QAT: val_ppl similar, but quant_ppl much worse- - Dropping pruning: val_ppl similar, but sparse_ppl worse- - Each component contributes — no single one is dominant--What this shows:- - The recipe is co-designed: each piece has a role- - No single component carries the result — it's the combination---### Exp 3: Matryoshka Accuracy-Efficiency Pareto [MUST-HAVE]--Setup:- - Single trained model with Matryoshka FFN- - Evaluate at FFN widths: full (2048), 2x (1024), 4x (512), 8x (256)- - Compare with separately trained models at each width- - Compare with knowledge distillation (full → small)--Metrics:- | Method | d_ff | Params | FLOPs | val_ppl | call_f1 |- |-----------------|------|--------|-------|---------|---------|- | Matryoshka full | 2048 | | | | |- | Matryoshka 2x | 1024 | | | | |- | Matryoshka 4x | 512 | | | | |- | Matryoshka 8x | 256 | | | | |- | Separate full | 2048 | | | | |- | Separate 2x | 1024 | | | | |- | Separate 4x | 512 | | | | |- | Separate 8x | 256 | | | | |- | Distilled 2x | 1024 | | | | |- | Distilled 4x | 512 | | | | |-- Plot: Pareto curve (FLOPs vs call_f1) for each method--Hypothesis:- - Matryoshka sub-models are within 1-3% of separately trained models- - Matryoshka beats distillation (no teacher-student mismatch)- - Single checkpoint, multiple operating points — simpler deployment--What this shows:- - Matryoshka is a practical elastic inference solution- - The accuracy-efficiency tradeoff is smooth (no cliff)- - One model replaces N separately trained models---### Exp 4: Optimizer Comparison [STRONG]--Setup:- - Muon+Adam (our split): Muon for kernels, Adam for rest- - Adam-only: AdamW with same schedule for all params- - SGD+momentum: baseline optimizer- - Same architecture, same data, same total steps--Metrics:- - Training loss curves (convergence speed)- - Final val_ppl and call_f1- - Wall-clock time per step--Hypothesis:- - Muon+Adam converges fastest (fewer steps to same loss)- - Adam-only reaches similar final quality but slower- - SGD is significantly worse (no adaptive rates for heterogeneous params)- - Muon adds negligible wall-clock overhead (Newton-Schulz is 5 matmuls)--What this shows:- - Muon accelerates convergence for attention/FFN matrices- - The split optimizer is justified: different params need different treatment---### Exp 5: QAT + Pruning Compression Analysis [STRONG]--Setup:- - Evaluate combinations:- Full precision, no pruning (baseline)- INT4 quantized (post-training, no QAT)- INT4 quantized (with QAT)- 50% pruned (no QAT)- INT4 + 50% pruned (both, with QAT)- - Report model size, inference speed, and accuracy--Metrics:- | Condition | Size(MB) | tok/s | val_ppl | call_f1 |- |----------------------|----------|-------|---------|---------|- | FP16 baseline | | | | |- | INT4 (no QAT) | | | | |- | INT4 (with QAT) | | | | |- | 50% pruned | | | | |- | INT4 + 50% pruned | | | | |-- Also: group_size sweep for QAT {16, 32, 64, 128}--Hypothesis:- - QAT recovers most of the INT4 quantization loss (quant_ppl gap < 0.5)- - Without QAT, INT4 has significant degradation at small model sizes- - Pruning + QAT compound: 8x size reduction with <5% accuracy loss- - Group size 32 is sweet spot (matches hardware vector width)--What this shows:- - QAT and pruning are essential for edge deployment- - Co-designed training (both at group_size=32) enables extreme compression- - Small models benefit more from QAT than large models (fewer redundant params)---### Exp 6: Activation Comparison (DReLU vs SwiGLU vs GeGLU) [STRONG]--Setup:- - Three activations, everything else identical- - Report: accuracy, activation sparsity, inference speed--Metrics:- | Activation | val_ppl | call_f1 | sparsity% | tok/s | quant_ppl |--Hypothesis:- - DReLU has highest activation sparsity (~65%)- - SwiGLU has best raw val_ppl (smoothest gradients)- - DReLU wins on call_f1 (sharper features for structured output)- - DReLU wins on tok/s (no sigmoid/erf, sparse activations)- - DReLU has best quant_ppl (sparse activations → better quantization)--What this shows:- - DReLU is the right choice for tool calling + edge deployment- - The sparsity-accuracy tradeoff favors DReLU in this domain---### Exp 7: Structured Loss Weight Sensitivity [STRONG]--Setup:- - Vary name weight: {1.0, 2.0, 3.0, 5.0, 10.0}- - Vary value weight: {1.0, 2.0, 3.0, 5.0}- - Key weight fixed at 1.5x--Metrics (heatmap: name_weight x value_weight):- - name_f1- - call_f1- - value_acc--Hypothesis:- - name_weight=1 (uniform): worst name_f1- - name_weight=3: sweet spot (sufficient signal without overwhelming)- - name_weight=10: name_f1 saturates, value_acc degrades (gradient- starved on value tokens)- - value_weight=2: good balance--What this shows:- - Structured weighting matters for structured output- - The weights should reflect the error impact hierarchy:- wrong name (catastrophic) > wrong value (incorrect) > wrong key (hallucinated)---### Exp 8: Training Curriculum Effect [NICE-TO-HAVE]--Setup:- - With curriculum: sort by tool count (easy → hard within epoch)- - Without curriculum: random shuffle- - Same total data, same epochs--Metrics:- - val_ppl convergence curve- - Per-tool-count call_f1 at epoch 1, 5, 10--Hypothesis:- - Curriculum helps early training (faster initial convergence)- - By end of training, gap narrows- - Biggest benefit on high-tool-count examples (model learns- simple patterns first, then extends to complex cases)--What this shows:- - Curriculum is useful but not critical- - Justifies the easy→hard ordering for tool-call training---### Exp 9: ZCRMSNorm vs Standard RMSNorm [NICE-TO-HAVE]--Setup:- - ZCRMSNorm (γ init to 0, applied as (1+γ)*x/RMS)- - Standard RMSNorm (γ init to 1, applied as γ*x/RMS)- - Same architecture, same recipe--Metrics:- - Training loss curves (first 1000 steps: stability)- - Final val_ppl and call_f1- - Gradient norm trajectory--Hypothesis:- - ZCRMSNorm has smoother early training (no initial activation spikes)- - Both converge to similar final quality- - ZCRMSNorm shows lower gradient norm variance in first 500 steps--What this shows:- - Zero-centering helps training stability, especially at small scale- - Free improvement: no added cost, just different initialization---### Priority--Must-have (core claims):- 1. Enc-dec vs dec-only (Exp 1) — justifies architecture choice- 2. Full recipe ablation (Exp 2) — proves co-design- 3. Matryoshka Pareto (Exp 3) — proves elastic inference--Strong additions:- 4. Optimizer comparison (Exp 4) — justifies Muon+Adam- 5. QAT + pruning compression (Exp 5) — deployment story- 6. Activation comparison (Exp 6) — justifies DReLU- 7. Loss weight sensitivity (Exp 7) — justifies structured weighting--Nice-to-have:- 8. Curriculum effect (Exp 8) — training efficiency- 9. ZCRMSNorm ablation (Exp 9) — initialization choice+ - We do not need MLPs for tiny models+ - Structured loss weighting targets the actual error distribution+ - QAT provides regularisation eliminating the need for dropout+ - Attention is a nonlinearity in the absence of MLP++----++Our blocks (with no_feedforward=True) are essentially:++ # encoder:+ x = x + Attention(Norm(x))++ # decoder:+ x = x + SelfAttn(Norm(x))+ x = x + CrossAttn(Norm(x))++The residual connection is the only pathway preserving information between layers.+Without FFN, there's no nonlinear feature transformation, each layer just incrementally mixes information via attention and adds it back.++The argument for Kimi-style residual bypass here would be: if you remove the residual, each attention layer gets to fully rewrite the representation rather than just adding a delta.+In a standard transformer with FFN, the FFN does the heavy "rewriting" work and the attention residual is fine as an additive refinement.+But in our attention-only setup, forcing attention into an additive role may be limiting, it can only nudge the representation, never reshape it.+Without FFN and without residuals, we lose the gradient highway entirely. Deep networks would struggle to train.+Kimi's technique likely works in models that still have FFN layers (or some other nonlinearity) providing an alternate signal path.++With the gated residual, each line becomes:++ # gated residual — each sublayer gate is a learnable scalar, initialized to 0+ # sigmoid(0) = 0.5, so training starts as half-strength standard residual+ # gate → 1 recovers the standard residual exactly++ # encoder (per layer)+ g = sigmoid(gate)+ x = x + g * Attention(Norm(x))++ # decoder (per layer)+ g_s = sigmoid(self_attn_gate)+ g_c = sigmoid(cross_attn_gate)++ x = x + g_s * SelfAttn(Norm(x))+ x = x + g_c * CrossAttn(Norm(x))++The full residual is always preserved. The gate only scales the sublayer contribution.+At init (gate=0, g=0.5): half-strength sublayer addition.+gate → 1 (g=1): standard residual. gate → -inf (g=0): identity, sublayer ignored.\ No newline at end of filediff --git a/notes/tool-selection.txt b/notes/tool-selection.txtindex f00f5dc..2b7dc3b 100644--- a/notes/tool-selection.txt+++ b/notes/tool-selection.txt@@ -15,12 +15,17 @@ select the right tool(s) for a given query. Two challenges:search_flights vs search_hotels vs search_restaurants). This is aretrieval problem, not a generation problem.+ (C) The model generates JSON tool calls autoregressively. While it learns+ JSON structure well (99-100% parse rate), it can still hallucinate+ invalid function names or argument keys. The valid tool definitions are+ already in the encoder input — we can constrain the decoder to only+ produce valid names/keys at inference time.+The standard approach is a separate retrieval model (e.g., embedding API)to pre-filter tools before passing them to the generator. This adds asecond model, a second inference pass, and a mismatch between therepresentations used for retrieval and generation.-## 2. Key Insight: Shared Encoder, Dual UseOur encoder already processes both queries and tool definitions duringdiff --git a/notes/tries.txt b/notes/tries.txtdeleted file mode 100644index 7dfed76..0000000--- a/notes/tries.txt+++ /dev/null@@ -1,267 +0,0 @@-Selective Trie-Constrained Decoding for Tool-Call Generation-=============================================================--## 1. Problem--The model generates JSON tool calls autoregressively. While it learns-JSON structure well (99-100% parse rate), it can still hallucinate-invalid function names or argument keys. The valid tool definitions are-already in the encoder input — we can constrain the decoder to only-produce valid names/keys at inference time.---## 2. Key Insight: Constrain What's Broken, Not What Works--The remaining errors are semantic, not syntactic:-- Error type Constrainable? Frequency- ───────────────────────── ────────────── ─────────- Wrong/hallucinated name YES (trie) Common- Hallucinated param key YES (trie) Common- Wrong argument values NO (open-ended) Common- Missing arguments NO Moderate- Malformed JSON NO (already 99%) Rare--Full grammar approaches (Outlines, llama.cpp GBNF, guidance) enforce-valid JSON on every token. This adds 10-30% overhead to solve a problem-that doesn't exist. Our approach only constrains the ~5% of tokens that-are inside name/key spans.---## 3. Architecture--Three components: a character-level trie, a JSON state machine, and-a token masking layer.--### 3a. Trie--Given tools: get_weather(location, unit), get_stock(symbol)--Name trie: Param trie (get_weather):-- (root) (root)- | / \- g l u- | | |- e o n- | | |- t c i- | | |- _ a t*- / \ |- w s t- | | |- e t i- | | |- a o o- | | |- t c n*- | |- h k* (* = terminal node)- |- e- |- r*--One name trie shared across all tools. One param trie per function,-built from its parameter keys.--### 3b. JSON State Machine--Tracks position in output to know when to constrain:-- ┌──────┐ "name":" ┌─────────┐ closing " ┌──────┐- │ FREE │────────────→│ IN_NAME │────────────→│ FREE │- └──────┘ └─────────┘ └──────┘- │ │- │ "arguments":{ then {" or ," │- │ ┌────────────┐ closing " │- └───────────────→│ IN_ARG_KEY │─────────────→┘- └────────────┘--State transitions driven by suffix matching on generated text.-Tracks nesting depth to avoid false triggers inside nested values.-Tracks string values with escape handling to avoid false triggers-inside string content.--### 3c. Token Masking--At each constrained step:-- 1. Look up current trie node (root + constrained_buf prefix)- 2. Gather valid first chars from trie children (+ closing " if terminal)- 3. For each candidate token (via TokenIndex first-char lookup):- - Walk trie char-by-char through token text- - If token contains ", check trie node is terminal at that point- - Mark as valid or invalid- 4. Set invalid token logits to -inf- 5. Fallback: if ALL tokens masked, revert to unconstrained + log warning--Example (IN_NAME state, after consuming "get_"):-- Trie node at "get_" has children: {w, s}- Scan TokenIndex for tokens starting with 'w' or 's':- "weather" → walk trie: w→e→a→t→h→e→r, all valid → ALLOW- "we" → walk trie: w→e, valid prefix → ALLOW- "stock" → walk trie: s→t→o→c→k, terminal → ALLOW- "sun" → walk trie: s→u, 'u' not child of 's' → BLOCK--Multi-char token boundary handling:- Token "ther\"" (closing quote mid-token):- Walk trie for 't','h','e','r' → valid path- Hit '"' → check if node is terminal → yes (weather*) → ALLOW- Correctly handles BPE tokens that span the name boundary.---## 4. Feedback Effect--The corrected token feeds back into the decoder buffer. All subsequent-generation conditions on the valid name:-- Without constraint: ..."name":"get_weathr","arguments":{"locaton":...- ~~~~~~~~~~~~ wrong ~~~~~ ~~~~~~~~ wrong ~~~~-- With constraint: ..."name":"get_weather","arguments":{"location":...- ~~~~~~~~~~~~ forced ~~~~ ~~~~~~~~~ model sees- correct name, generates- better arguments--This is why in-place correction beats post-processing — the decoder-sees correct context when generating arguments.---## 5. Comparison with Full Grammar Approaches-- ┌─────────────────────────────────────────────────────────┐- │ Full grammar (CFG/PDA) │- │ │- │ Constrains EVERY token: brackets, colons, commas, │- │ string delimiters, value types, nesting, whitespace. │- │ Guarantees syntactically valid JSON always. │- │ │- │ Cost: FSM/PDA transition on every token, large state │- │ space, precomputed token masks per state, high memory. │- │ Typical overhead: 10-30% slowdown on all tokens. │- └─────────────────────────────────────────────────────────┘-- ┌─────────────────────────────────────────────────────────┐- │ Selective constraint (trie + state machine) │- │ │- │ Only constrains tokens inside "name":"___" and │- │ argument key "___" spans. Everything else │- │ (structure, values, punctuation) is unconstrained. │- │ │- │ Cost: lightweight state machine tracks JSON position │- │ via suffix matching. Trie walk + token masking only │- │ activates during name/key spans (~5% of tokens). │- │ Zero overhead on remaining ~95% of tokens. │- └─────────────────────────────────────────────────────────┘--Why full grammar is overkill here:- - 99-100% JSON parse rate means structure is already learned- - Full grammar pays per-token cost for ~0% parse improvement- - Can force the model into corners where the most likely valid- continuation is nonsensical (distorts value generation)--Why selective works:- - Targets the actual failure mode (hallucinated names/keys)- - Near-zero cost (only active on ~5% of tokens)- - Never distorts value generation- - Corrected tokens improve downstream generation via feedback---## 6. Implementation Details-- token_strings / TokenIndex: cached per tokenizer (built once)- Device-to-host transfer: only when state machine is active- (IN_NAME or IN_ARG_KEY); FREE state stays on jnp.argmax- Enabled by default; disable with --no-constrained---## 7. Experiment Design--All trie experiments use our val set (5000 samples) — clean controlled-variable where only the decoding mode changes. Exact reference calls-give precise name_f1, param_halluc, value_acc. Per-tool-count breakdown-already implemented. Reproducible with a seed.--### Exp 1: Ablation — Constrained vs Unconstrained [MUST-HAVE]--Same checkpoint, same val set, three conditions:- (a) Unconstrained (--no-constrained)- (b) Name-only constraint (trie on function names, keys unconstrained)- (c) Full selective constraint (trie on names + argument keys)-- | Condition | name_f1 | call_f1 | exact_match | param_halluc | param_miss | value_acc | json_parse |--Expects: name_f1 improves (a→b), param_halluc drops (b→c), json_parse-unchanged across all three (confirms structure isn't the problem).-Needs a "names_only" mode flag in ConstrainedDecoder.--### Exp 2: Latency Overhead [MUST-HAVE]--500 samples, wall-clock time and tok/s, three conditions, 3 runs each.-- | Condition | tok/s | Overhead |--Also report: % tokens where constraint was active, avg time in-apply_constraints per constrained token.-Expects: near-zero overhead because constraints activate on ~5% of tokens.--### Exp 3: Error Analysis — What Constraints Fix [MUST-HAVE]--Run constrained and unconstrained on same 1000 samples. Categorize:- (a) Both correct → constraint irrelevant- (b) Unconstrained wrong, constrained correct → constraint helped- (c) Both wrong (same error) → constraint irrelevant- (d) Both wrong (different error) → constraint changed but didn't fix- (e) Unconstrained correct, constrained wrong → constraint hurt (should be ~0)--For category (b), sub-classify:- - Misspelled function name (get_weathr → get_weather)- - Hallucinated function name (fetch_weather → get_weather)- - Hallucinated parameter key (city → location)- - Misspelled parameter key (locaton → location)--Present as sankey diagram + table of top-10 fixes with before/after.-Category (e) should be empty — proves constraints are safe.--### Exp 4: Breakdown by Tool Count [STRONG]--Use existing per_count breakdown (1-10 tools per example).-Grouped bar chart: name_f1 and param_halluc per bucket, constrained vs not.--Expects: more tools → more names to confuse → bigger constraint benefit.-Shows constraint value scales with schema complexity.--### Exp 5: Downstream Effect — Name Correction Improves Arguments [STRONG]--Focus on examples where unconstrained gets the name wrong.-Compare value_acc and args_acc when name is wrong vs corrected.--Expects: corrected name → better arguments (feedback effect).-Justifies in-place correction over post-processing.--### Exp 6: Benefit Across Training Epochs [STRONG]--Checkpoints at epochs 1, 3, 5, 7, 10. Each evaluated with and without.-Line plot: call_f1 constrained vs unconstrained across epochs.--Expects: biggest delta mid-training ("almost right" phase), positive-everywhere. Shows constraints are complementary to training.--### Exp 7: Full Grammar Comparison [NICE-TO-HAVE]--Implement Outlines-style full JSON grammar constraint as baseline.-Three-way comparison: unconstrained, full grammar, selective trie.--Expects: full grammar has ~same call_f1 but much higher overhead.-Strongest competitive claim but most implementation work.--### Exp 8: Out-of-Distribution Tool Schemas [NICE-TO-HAVE]--Synthetic tools with names/keys not in training data (20-50 examples).-Constrained vs unconstrained.--Expects: OOD names have highest hallucination rate → biggest delta.-Shows value for custom/long-tail APIs. Needs manual data creation.diff --git a/notes/voice-to-action.txt b/notes/voice-to-action.txtnew file mode 100644index 0000000..e69de29diff --git a/src/cli.py b/src/cli.pyindex 1b4a018..91a55e4 100644--- a/src/cli.py+++ b/src/cli.py@@ -15,7 +15,7 @@ def main():p = sub.add_parser("train", add_help=False)p.add_argument("--checkpoint", type=str, default=None)- p.add_argument("--epochs", type=int, default=3)+ p.add_argument("--epochs", type=int, default=1)p.add_argument("--batch-size", type=int, default=32)p.add_argument("--lr", type=float, default=3e-4)p.add_argument("--muon-lr", type=float, default=0.02)@@ -46,7 +46,7 @@ def main():p.add_argument("--activation", type=str, default="swiglu", choices=["drelu", "swiglu", "geglu"])p.add_argument("--num-memory-slots", type=int, default=128,help="(DEPRECATED — ignored, kept for checkpoint compat)")- p.add_argument("--mat-factors", type=int, nargs="*", default=[2, 4, 8],+ p.add_argument("--mat-factors", type=int, nargs="*", default=[2, 4],help="Matryoshka FFN shrink factors, e.g. 2=half width (default: 2 4)")p.add_argument("--dropout", type=float, default=0.0,help="Dropout rate for residual connections (default: 0.0)")@@ -58,6 +58,8 @@ def main():help="Dimension of contrastive projection head (default: 128)")p.add_argument("--conv-kernel-size", type=int, default=0,help="Conformer conv kernel size in encoder (0=disabled, 15=default when enabled)")+ p.add_argument("--no-feedforward", action=argparse.BooleanOptionalAction, default=True,+ help="Remove feedforward layers entirely (default: True)")p = sub.add_parser("tokenize", add_help=False)p.add_argument("--max-samples", type=int, default=None,diff --git a/src/data.py b/src/data.pyindex 48d114c..d09a363 100644--- a/src/data.py+++ b/src/data.py@@ -380,8 +380,8 @@ def load_tool_calls(split="train", max_samples=None, return_global_indices=Falseperm = rng.permutation(n)val_size = min(5000, int(n * 0.1))- val_indices = perm[:val_size]- train_indices = perm[val_size:]+ val_indices = perm[-val_size:]+ train_indices = perm[:-val_size]if split in ("validation", "val", "test"):global_indices = val_indices.astype(np.int64)diff --git a/src/eval.py b/src/eval.pyindex 9319a48..c9aeb79 100644--- a/src/eval.py+++ b/src/eval.py@@ -345,7 +345,7 @@ def benchmark_tool_calls(model, params, tokenizer, num_samples=200, max_gen_len=}-def benchmark_retrieval(model, params, tokenizer, num_samples=500, max_len=256, ks=(1, 3, 5, 10), ds=None):+def benchmark_retrieval(model, params, tokenizer, num_samples=500, max_len=256, ks=(1, 2, 3, 4, 5), ds=None):"""Benchmark contrastive retrieval: Recall@k and MRR over validation set.For each query, ranks all tools from that example by cosine similaritydiff --git a/src/model.py b/src/model.pyindex df3f547..9e34ea3 100644--- a/src/model.py+++ b/src/model.py@@ -50,6 +50,7 @@ class TransformerConfig:contrastive_dim: int = 128conv_kernel_size: int = 0enable_speech: bool = False+ no_feedforward: bool = True@propertydef jax_dtype(self):@@ -221,7 +222,7 @@ class ConvModule(nn.Module):class EncoderBlock(nn.Module):- """Pre-norm self-attention + pre-norm FFN with residual connections."""+ """Pre-norm self-attention + pre-norm FFN with gated residual connections."""num_heads: intnum_kv_heads: intd_model: int@@ -231,27 +232,32 @@ class EncoderBlock(nn.Module):activation: str = "drelu"dropout_rate: float = 0.0conv_kernel_size: int = 0+ no_feedforward: bool = True@nn.compactdef __call__(self, x, mask=None, rope=None, ffn_mask=None, deterministic=True):+ gate = nn.sigmoid(self.param("attn_gate", jinit.zeros, ())).astype(self.dtype)residual = xx = ZCRMSNorm(dtype=self.dtype)(x)x = MultiHeadAttention(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers, self.dtype, name="self_attn")(x, x, mask=mask, rope=rope)- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual+ x = residual + gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)if self.conv_kernel_size > 0:+ conv_gate = nn.sigmoid(self.param("conv_gate", jinit.zeros, ())).astype(self.dtype)residual = xx = ZCRMSNorm(dtype=self.dtype)(x)x = ConvModule(self.d_model, self.conv_kernel_size, self.num_layers,self.dtype, name="conv_module")(x)- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual+ x = residual + conv_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)- residual = x- x = ZCRMSNorm(dtype=self.dtype)(x)- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual+ if not self.no_feedforward:+ ffn_gate = nn.sigmoid(self.param("ffn_gate", jinit.zeros, ())).astype(self.dtype)+ residual = x+ x = ZCRMSNorm(dtype=self.dtype)(x)+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)+ x = residual + ffn_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)return x@@ -267,6 +273,7 @@ class _EncoderScanBody(nn.Module):activation: str = "drelu"dropout_rate: float = 0.0conv_kernel_size: int = 0+ no_feedforward: bool = Truedeterministic: bool = True@nn.compact@@ -275,7 +282,7 @@ class _EncoderScanBody(nn.Module):x = EncoderBlock(self.num_heads, self.num_kv_heads, self.d_model, self.d_ff,self.num_layers, self.dtype, self.activation, self.dropout_rate,- self.conv_kernel_size,+ self.conv_kernel_size, self.no_feedforward,)(x, mask, rope, ffn_mask, self.deterministic)return (x, mask, rope, ffn_mask), None@@ -299,7 +306,7 @@ class Encoder(nn.Module):(x, _, _, _), _ = ScanBlock(cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff,cfg.total_layers, dt, cfg.activation, cfg.dropout_rate,- cfg.conv_kernel_size, deterministic, name="layers",+ cfg.conv_kernel_size, cfg.no_feedforward, deterministic, name="layers",)((x, mask, rope, ffn_mask), None)x = ZCRMSNorm(dtype=dt, name="final_norm")(x)@@ -315,27 +322,32 @@ class DecoderBlock(nn.Module):dtype: jnp.dtype = jnp.bfloat16activation: str = "drelu"dropout_rate: float = 0.0+ no_feedforward: bool = True@nn.compactdef __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, ffn_mask=None, deterministic=True):+ self_gate = nn.sigmoid(self.param("self_attn_gate", jinit.zeros, ())).astype(self.dtype)residual = xx = ZCRMSNorm(dtype=self.dtype)(x)x = MultiHeadAttention(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers, self.dtype, name="self_attn")(x, x, mask=self_mask, rope=rope)- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual+ x = residual + self_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)+ cross_gate = nn.sigmoid(self.param("cross_attn_gate", jinit.zeros, ())).astype(self.dtype)residual = xx = ZCRMSNorm(dtype=self.dtype)(x)x = MultiHeadAttention(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers, self.dtype, name="cross_attn")(x, encoder_out, mask=cross_mask)- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual+ x = residual + cross_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)- residual = x- x = ZCRMSNorm(dtype=self.dtype)(x)- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual+ if not self.no_feedforward:+ ffn_gate = nn.sigmoid(self.param("ffn_gate", jinit.zeros, ())).astype(self.dtype)+ residual = x+ x = ZCRMSNorm(dtype=self.dtype)(x)+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)+ x = residual + ffn_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)return x@@ -351,6 +363,7 @@ class _DecoderScanBody(nn.Module):dtype: jnp.dtype = jnp.bfloat16activation: str = "drelu"dropout_rate: float = 0.0+ no_feedforward: bool = Truedeterministic: bool = True@nn.compact@@ -359,6 +372,7 @@ class _DecoderScanBody(nn.Module):x = DecoderBlock(self.num_heads, self.num_kv_heads, self.d_model, self.d_ff,self.num_layers, self.dtype, self.activation, self.dropout_rate,+ self.no_feedforward,)(x, encoder_out, self_mask, cross_mask, rope, ffn_mask, self.deterministic)return (x, encoder_out, self_mask, cross_mask, rope, ffn_mask), None@@ -381,7 +395,7 @@ class Decoder(nn.Module):(x, _, _, _, _, _), _ = ScanBlock(cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff,cfg.total_layers, dt, cfg.activation, cfg.dropout_rate,- deterministic, name="layers",+ cfg.no_feedforward, deterministic, name="layers",)((x, encoder_out, self_mask, cross_mask, rope, ffn_mask), None)x = ZCRMSNorm(dtype=dt)(x)diff --git a/src/train.py b/src/train.pyindex 8f8c24b..dcd861d 100644--- a/src/train.py+++ b/src/train.py@@ -531,6 +531,7 @@ def train(args):contrastive_dim=getattr(args, "contrastive_dim", 128),conv_kernel_size=getattr(args, "conv_kernel_size", 0),enable_speech=getattr(args, "enable_speech", False),+ no_feedforward=getattr(args, "no_feedforward", True),)global _GROUP_SIZE, _MAT_FACTORS, _MAT_FF_WIDTHS, _D_FF, _CONTRASTIVE_WEIGHT@@ -538,7 +539,7 @@ def train(args):_CONTRASTIVE_WEIGHT = getattr(args, "contrastive_weight", 0.1)_D_FF = config.d_ffmat_factors_raw = getattr(args, "mat_factors", None)- if mat_factors_raw:+ if mat_factors_raw and not config.no_feedforward:_MAT_FACTORS = tuple(f for f in mat_factors_raw if f > 1)_MAT_FF_WIDTHS = tuple(config.d_ff // f for f in _MAT_FACTORS)else:@@ -971,6 +972,9 @@ def train(args):_gen_model = eval_model_gen_label = "full"+ _mat_params = _quantize_params(_mat_params, group_size=_GROUP_SIZE)+ _gen_label += " INT4"+all_preds = []_gen_t0 = time.perf_counter()for _ei in tqdm(range(0, len(all_eval_examples), _EVAL_BATCH),