Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

No FFN by default

HenryNdubuakubc3a3ddf3767
diff --git a/notes/architecture.txt b/notes/architecture.txt
index c421403..a8e4a91 100644
--- a/notes/architecture.txt
+++ b/notes/architecture.txt
@@ -10,711 +10,48 @@ structured weighting + QAT + pruning) that produces a small model
competitive with much larger decoder-only models on tool calling.
The architecture choices are interdependent:
- - Encoder-decoder enables shared-encoder retrieval (see tool-selection.txt)
- - Matryoshka FFN enables elastic deployment at multiple compute budgets
- - Structured loss weighting targets the actual error distribution
- - QAT + pruning enable deployment on edge devices
- - Constrained decoding exploits known schemas (see tries.txt)
-
-Each piece alone is known. The system is the contribution.
-
-
-## 2. Architecture
-
-### 2a. Why Encoder-Decoder (Not Decoder-Only)
-
-Almost all tool-calling models are decoder-only. We use encoder-decoder
-because tool calling has a natural asymmetry:
-
- Input (encode once): Output (generate):
- ┌──────────────────────┐ ┌──────────────────────┐
- │ Query: "What's the │ │[{"name":"get_weather"│
- │ weather in SF?" │ │ "arguments": │
- │ <tools> │ │ {"location":"SF"}} │
- │ [{"name":"get_weather│ │ ] │
- │ "parameters":{...}}│ └──────────────────────┘
- │ {"name":"get_stock" │
- │ "parameters":{...}}│
- │ ] │
- └──────────────────────┘
-
- Encoder input = context (query + tool definitions) — processed once
- Decoder output = structured generation (JSON tool calls) — autoregressive
-
-Benefits:
- (a) The encoder processes all tools in parallel via bidirectional
- self-attention. A decoder-only model processes tools left-to-right,
- which means later tools attend to earlier tools but not vice versa.
- For discriminating between semantically similar tools (search_flights
- vs search_hotels), bidirectional context helps.
-
- (b) Encoder output is reused across all decoder steps via cross-attention.
- In a decoder-only model, KV cache stores the full context, but the
- representation is computed left-to-right with causal masking — no
- opportunity for the context to self-refine bidirectionally.
-
- (c) The encoder can serve double duty: cross-attention for generation
- AND contrastive retrieval for tool selection (see tool-selection.txt).
- A decoder-only model would need a separate retrieval module.
-
- (d) Smaller parameter budget. With the same d_model and layer count,
- an encoder-decoder can encode context more efficiently because
- the encoder layers don't need causal masking, cross-attention
- projections, or autoregressive logic.
-
-
-### 2b. Zero-Centered RMSNorm (ZCRMSNorm)
-
-Standard RMSNorm: γ * x / RMS(x), with γ initialized to 1.
-Our ZCRMSNorm: (1 + γ) * x / RMS(x), with γ initialized to 0.
-
- class ZCRMSNorm(nn.Module):
- def __call__(self, x):
- scale = self.param("scale", zeros, (x.shape[-1],))
- rms = sqrt(mean(x^2, axis=-1, keepdims=True) + eps)
- return (1 + scale) * x / rms
-
-At initialization, (1 + 0) * x / RMS(x) = x / RMS(x) — the layer
-normalizes but applies no learned scaling. This is similar in spirit
-to ReZero (Bachlechner et al. 2020) and fixup initialization: at
-init, each sub-layer's contribution is minimal, and the residual
-stream dominates. The network starts as a near-identity function
-and gradually learns to use each layer.
-
-Why this matters for small models:
- - Small models are more sensitive to initialization. Standard RMSNorm
- with γ=1 can cause early training instability because pre-norm
- layers immediately rescale activations.
- - Zero-init means gradients flow cleanly through the residual stream
- at the start of training, avoiding the "early chaos" period.
-
-Applied everywhere: encoder pre-norm, decoder pre-norm, QK-norm,
-ConvModule norm. Consistent zero-centering throughout.
-
-
-### 2c. Grouped Query Attention (GQA) with QK-Norm
-
-Standard multi-head attention with grouped KV heads:
-
- Q: (B, num_heads, T, head_dim) — full heads
- K: (B, num_kv_heads, T, head_dim) — grouped (repeated for attention)
- V: (B, num_kv_heads, T, head_dim) — grouped
-
- num_heads=8, num_kv_heads=2 → 4:1 ratio, 75% KV parameter savings
-
-QK-norm: apply ZCRMSNorm to Q and K after projection, before attention.
-
- q = ZCRMSNorm(q_proj(x))
- k = ZCRMSNorm(k_proj(x))
- attn = softmax(q @ k^T / sqrt(d_head))
-
-QK-norm prevents attention logit growth as training progresses (the
-"logit drift" problem observed in large-scale training). With zero-init
-normalization, Q and K start with unit-scale dot products and gradually
-learn scale — stable from step 1.
-
-This combination (GQA + QK-norm + ZCRMSNorm) is individually known
-(Gemma 2, ViT-22B use QK-norm; GQA from Ainslie et al. 2023) but the
-zero-centered variant on Q/K is our specific choice.
-
-
-### 2d. DReLU Activation
-
-Three activation options: SwiGLU, GeGLU, DReLU. Default is DReLU.
-
- SwiGLU: silu(gate) * up — standard, smooth
- GeGLU: gelu(gate) * up — smoother, slightly more expensive
- DReLU: relu(gate) * relu(up) — simplest, sparsest
-
- h = relu(W_gate @ x) * relu(W_up @ x)
- out = W_down @ h
-
-DReLU applies ReLU to both gate and up projections independently.
-This produces the sparsest activations: any input where either gate
-OR up projection is negative gets zeroed. Empirically, ~60-70% of
-hidden units are zero at any given step.
-
-Why DReLU for tool calling:
- - Activation sparsity enables hardware-level speedups (skip zero
- multiplications). Critical for edge deployment.
- - Works synergistically with magnitude pruning: sparse activations
- mean many weights have low magnitude → cleaner pruning signal.
- - Cheaper than SwiGLU/GeGLU: no sigmoid/erf computation.
- - For structured output (JSON), the model doesn't need smooth
- gradients for nuanced language — it needs sharp, discriminative
- features for selecting the right tool name and argument keys.
-
-
-### 2e. Matryoshka FFN
-
-Train a single model that contains multiple sub-models at different
-FFN widths. At inference, choose the width based on compute budget.
-
- Full model: d_ff = 2048 (all neurons active)
- 2x sub-model: d_ff = 1024 (first 1024 neurons)
- 4x sub-model: d_ff = 512 (first 512 neurons)
-
-Implementation: prefix masking on FFN hidden dimension.
-
- def __call__(self, x, ffn_mask=None):
- gate = gate_proj(x) # (B, T, d_ff)
- up = up_proj(x) # (B, T, d_ff)
- h = relu(gate) * relu(up)
- if ffn_mask is not None:
- h = h * ffn_mask # zero out neurons beyond width
- return down_proj(h)
-
-During training, each batch is split across widths:
-
- Batch (B=32):
- items 0-10: full width (d_ff=2048) — all neurons
- items 11-21: 2x width (d_ff=1024) — first half
- items 22-31: 4x width (d_ff=512) — first quarter
-
- All items share the same forward pass (single jit-compiled function).
- The ffn_mask varies per item. No separate models, no distillation.
-
-The prefix constraint is key: sub-models always use the FIRST k neurons.
-This means the first 512 neurons must be useful on their own, the first
-1024 must be useful, etc. The training objective forces the model to
-organize information by importance along the FFN dimension — most
-important features in early neurons.
-
-At inference:
- - Edge device: use 4x sub-model (512 neurons, ~25% FFN compute)
- - Server: use full model (2048 neurons, full quality)
- - Same checkpoint, same parameters, different masks
-
-Why not distillation:
- - Distillation requires training N separate models
- - Matryoshka trains one model, gets N sub-models
- - Sub-models share attention weights — only FFN width changes
- - Parameter estimation for any sub-model is straightforward
-
-
-### 2f. Shared Encoder with Contrastive Head
-
-See tool-selection.txt for full details. Summary:
-
- Encoder → mean pool → Dense(d_model, 128) → L2 normalize → cosine similarity
-
- Same encoder serves:
- (a) Cross-attention in decoder (generation)
- (b) Contrastive retrieval (tool selection from large catalogs)
-
- Adds <0.01% parameters (one 512x128 matrix + learnable temperature).
-
-
-### 2g. Conformer Convolution Module (Optional)
-
-When conv_kernel_size > 0, each encoder block adds a convolution module
-between self-attention and FFN:
-
- self-attn → ConvModule → FFN
-
- ConvModule:
- pointwise up (d → 2d) → GLU gate → depthwise conv (k=31) →
- ZCRMSNorm → pointwise down (d → d)
-
-This is the Conformer (Gulati et al. 2020) pattern. The depthwise
-convolution captures local patterns (token n-grams in tool definitions,
-local acoustic patterns in speech). Complementary to global self-attention.
-
-Primary motivation: speech pathway benefits significantly from local
-convolution. For text-only, conv_kernel_size=0 (disabled).
-
-
-### 2h. Speech Pathway
-
-Optional speech input via FastConformer-style 8x subsampling:
-
- Input: (B, T, 80) mel spectrogram
- Output: (B, T//8, d_model) features fed to shared encoder
-
- Subsampling (3 stages of stride-2 = 8x total):
- Conv2D(1→C, k=3, s=2) + ReLU
- DW-Conv(C, k=3, s=2) + PW-Conv(C→C) + ReLU
- DW-Conv(C, k=3, s=2) + PW-Conv(C→C) + ReLU
- Reshape + Linear → d_model
-
- SpecAugment (training only):
- 2 time masks (max 100 frames)
- 1 frequency mask (max 27 bins)
-
-The speech subsystem converts audio to the same representation space
-as text. After subsampling, the encoder processes speech features
-identically to text embeddings — same attention, same FFN, same
-contrastive head. Unified encoder for both modalities.
-
-
-### 2i. Memory-Efficient Layer Stacking
-
-Both encoder and decoder use nn.scan with remat:
-
- ScanBlock = nn.scan(
- nn.remat(BlockBody),
- variable_axes={"params": 0}, # stack params along axis 0
- split_rngs={"params": True},
- length=num_layers,
- )
-
- - nn.scan: shares the block's Python code across layers, stacking
- parameters as (num_layers, ...) tensors. Reduces compilation time.
- - nn.remat: recomputes activations during backward pass instead of
- storing them. Trades ~30% more compute for ~60% less memory.
-
-For a small model this matters less, but it enables scaling experiments
-with deeper models on the same hardware.
-
-
-## 3. Training Recipe
-
-### 3a. Muon + Adam Split Optimizer
-
-Not all parameters benefit from the same optimizer. We split:
-
- Muon optimizer: all Dense kernels (2D+ tensors)
- - Orthogonalize gradients via Newton-Schulz polar decomposition
- - Nesterov momentum (μ=0.95)
- - Weight decay 0.01
- - Separate WSD learning rate schedule
-
- Adam optimizer: everything else (embeddings, norms, biases, 1D params)
- - AdamW with β2=0.95 (lower than default 0.999 for stability)
- - No weight decay
- - Separate WSD learning rate schedule
-
-Muon's gradient orthogonalization:
-
- def _newton_schulz(G, steps=5):
- a, b, c = 3.4445, -4.7750, 2.0315
- X = G / norm(G)
- for _ in range(steps):
- A = X @ X^T
- B = b*A + c*A@A
- X = a*X + B@X
- return X
-
- This approximates the polar decomposition G = U*S*V^T and returns U*V^T.
- The gradient is projected onto the nearest orthogonal matrix.
-
-Why Muon for attention/FFN kernels:
- - Orthogonalized updates prevent weight matrix collapse (rank
- deficiency) in small models where d_model is limited.
- - Newton-Schulz is cheap: 5 iterations of matrix multiply, no SVD.
- - Empirically, Muon converges faster than Adam on projection matrices.
-
-Why Adam for the rest:
- - Embeddings are lookup tables — orthogonalization doesn't apply.
- - Norm scales are 1D — orthogonalization is meaningless.
- - Adam's adaptive learning rates are better for heterogeneous params.
-
-
-### 3b. Warmup-Stable-Decay (WSD) Schedule
-
- ┌─────────────────────────────────────────────────────────┐
- │ LR │
- │ ▲ │
- │ │ ┌──────────────────────────────┐ │
- │ │ /│ stable │\ │
- │ │ / │ │ \ cosine │
- │ │ / │ │ \ decay │
- │ │ / │ │ \ │
- │ │ / │ │ \____ │
- │ │/ warmup │ │
- │ └──────┴──────────────────────────────┴────────→ step │
- │ 0% ~5% 85% 100% │
- └─────────────────────────────────────────────────────────┘
-
- - Warmup: linear ramp from 0 to peak (first ~5% of steps)
- - Stable: hold at peak (majority of training)
- - Decay: cosine anneal to 5% of peak (last 15% of steps)
-
-Applied separately to Adam and Muon with different peak values.
-WSD is simpler than cosine-with-restarts and matches or beats it
-for fixed-budget training (MiniCPM, Zhai et al. 2022).
-
-
-### 3c. Quantization-Aware Training (QAT)
-
-Every 100 steps, apply fake INT4 quantization to all Dense kernels:
-
- def _fake_quantize_int4(w, group_size=32):
- # Group along input dimension
- w_grouped = w.reshape(num_groups, group_size, out_feat)
- # Per-group symmetric quantization
- scale = max(abs(w_grouped), axis=1) / 7.0
- w_q = clip(round(w_grouped / scale), -8, 7) * scale
- # Straight-through estimator
- return w + stop_gradient(w_q - w)
-
- - Symmetric INT4: 4-bit integers [-8, 7] with per-group scale
- - Group size 32: each 32 consecutive input features share a scale
- - STE: forward uses quantized weights, backward uses real gradients
- - Applied conditionally: only every 100 steps (reduces overhead)
-
-The model learns to be robust to INT4 quantization during training.
-At deployment, weights are actually quantized to INT4 → 4x memory
-reduction, hardware-accelerated INT4 matmul on supporting devices.
-
-Why every 100 steps instead of every step:
- - QAT adds ~15% overhead per step (extra quantize + forward pass)
- - Every 100 steps: <0.15% amortized overhead
- - Weights change slowly — quantization noise from step N is still
- representative at step N+99
- - Same rationale as contrastive loss frequency (see tool-selection.txt)
-
-
-### 3d. Group-Aligned Gradual Pruning
-
-See sparsification.txt for full details. Summary:
-
- - Group-wise magnitude pruning with group_size=32 — same groups as QAT
- - Cubic gradual schedule: 0% → target over middle third of epoch
- - Global threshold across all layers and matrices
- - Mask applied after gradient update, locked after pruning epoch
-
- The critical co-design: pruned groups are exactly the groups the INT4
- quantization kernel operates on. A pruned group = zero weights + zero
- scale factor = zero memory loads + zero MACs. On edge INT4 kernels,
- 50% group sparsity → 50% fewer memory accesses and 50% fewer compute
- operations, directly translating to battery savings.
-
-
-### 3e. Structured Per-Token Loss Weighting
-
-Not all tokens in the decoder output are equally important:
-
- [{"name":"get_weather","arguments":{"location":"San Francisco"}}]
-
- Token role Weight Rationale
- ────────── ────── ─────────
- name value 3.0 Wrong name = completely wrong call
- arg value 2.0 Wrong value = wrong behavior
- arg key 1.5 Wrong key = hallucinated parameter
- structure 1.0 JSON syntax (already learned well)
-
-Implemented as per-token mask in loss computation:
-
- loss = sum(CE(logits, targets) * token_weights) / sum(token_weights)
-
-The weighting is computed during data preparation by parsing the
-target JSON and labeling each token's role. This focuses gradient
-signal on the tokens that matter most for tool-call correctness.
-
-Why not uniform weighting:
- - 60-70% of decoder tokens are JSON structure ({, }, [, ], :, ", ,)
- - The model learns structure quickly (99%+ parse rate)
- - Under uniform weighting, gradient signal is dominated by structure
- tokens — the model keeps "improving" on tokens it already gets right
- - 3x weight on names means 3x gradient signal for the critical tokens
-
-
-### 3f. Z-Loss Regularization
-
-Small auxiliary loss on logit magnitude:
-
- z_loss = 1e-4 * mean(logsumexp(logits, axis=-1)^2)
-
-Prevents logits from growing unboundedly during training. Without this,
-logits can reach 100+ in magnitude, causing numerical issues in softmax
-and making the model overconfident. Z-loss keeps logits in a reasonable
-range (~10-20) without affecting the argmax (relative ordering preserved).
-
-
-### 3g. Multi-Objective Training Schedule
-
-All objectives in a single training loop, but at different frequencies:
-
- Every step:
- - Text generation loss (CE + z-loss)
- - Matryoshka FFN masking (batch split across widths)
- - Gradient clip (global norm 1.0)
- - Muon/Adam update
-
- Every 100 steps:
- - QAT (fake INT4 quantization)
- - Contrastive loss (CLIP-style, weight 0.1)
-
- During pruning epoch (steps 33%-67%):
- - Every prune_interval steps: recompute prune mask
- - Apply mask after each gradient update
-
- End of pruning epoch:
- - Lock mask, report final sparsity
-
-This schedule minimizes overhead: the expensive objectives (QAT,
-contrastive) run infrequently, and their signals are low-noise
-enough that per-step computation is unnecessary.
-
-
-## 4. Experiment Design
-
-### Exp 1: Encoder-Decoder vs Decoder-Only [MUST-HAVE]
-
-Setup:
- - Train encoder-decoder (our architecture) and decoder-only
- (same total params, same d_model, combined layer count) on
- identical data with identical recipe
- - Decoder-only: concatenate [query | tools | answer] as single
- sequence, causal attention throughout
- - Match total FLOPs as closely as possible
-
-Metrics:
- | Architecture | Params | name_f1 | call_f1 | exact_match | value_acc |
-
- Also report:
- - Inference latency (tok/s) for both
- - Encoder reuse efficiency (for enc-dec: encode once, decode many)
-
-Hypothesis:
- - Enc-dec should win on name_f1 (bidirectional tool encoding)
- - Enc-dec should win on multi-tool examples (better tool discrimination)
- - Dec-only may be competitive on single-tool examples
- - Enc-dec inference faster when generating multiple calls per context
- (encoder output cached, decoder runs multiple times)
-
-What this shows:
- - The architectural choice is justified for tool calling
- - Bidirectional encoding of tool definitions helps discrimination
-
-
-### Exp 2: Full Recipe Ablation [MUST-HAVE]
-
-Setup:
- - Full recipe: all components enabled
- - Drop-one ablations (each removes exactly one component):
- (a) No Muon (Adam-only for all params)
- (b) No QAT (no fake quantization)
- (c) No pruning (no magnitude pruning)
- (d) No structured loss weighting (uniform weights)
- (e) No Matryoshka (full FFN width only)
- (f) No contrastive (already in tool-selection.txt Exp 2)
- (g) No z-loss
- (h) No DReLU (use SwiGLU instead)
-
-Metrics:
- | Condition | val_ppl | name_f1 | call_f1 | quant_ppl | sparse_ppl |
- |---------------------|---------|---------|---------|-----------|------------|
- | Full recipe | | | | | |
- | - Muon | | | | | |
- | - QAT | | | | N/A | |
- | - Pruning | | | | | N/A |
- | - Struct. weights | | | | | |
- | - Matryoshka | | | | | |
- | - z-loss | | | | | |
- | - DReLU (+SwiGLU) | | | | | |
-
-Hypothesis:
- - Full recipe is best on aggregate
- - Dropping structured weighting hurts name_f1 most (name tokens
- lose 3x gradient signal)
- - Dropping Muon hurts convergence speed (need more epochs)
- - Dropping QAT: val_ppl similar, but quant_ppl much worse
- - Dropping pruning: val_ppl similar, but sparse_ppl worse
- - Each component contributes — no single one is dominant
-
-What this shows:
- - The recipe is co-designed: each piece has a role
- - No single component carries the result — it's the combination
-
-
-### Exp 3: Matryoshka Accuracy-Efficiency Pareto [MUST-HAVE]
-
-Setup:
- - Single trained model with Matryoshka FFN
- - Evaluate at FFN widths: full (2048), 2x (1024), 4x (512), 8x (256)
- - Compare with separately trained models at each width
- - Compare with knowledge distillation (full → small)
-
-Metrics:
- | Method | d_ff | Params | FLOPs | val_ppl | call_f1 |
- |-----------------|------|--------|-------|---------|---------|
- | Matryoshka full | 2048 | | | | |
- | Matryoshka 2x | 1024 | | | | |
- | Matryoshka 4x | 512 | | | | |
- | Matryoshka 8x | 256 | | | | |
- | Separate full | 2048 | | | | |
- | Separate 2x | 1024 | | | | |
- | Separate 4x | 512 | | | | |
- | Separate 8x | 256 | | | | |
- | Distilled 2x | 1024 | | | | |
- | Distilled 4x | 512 | | | | |
-
- Plot: Pareto curve (FLOPs vs call_f1) for each method
-
-Hypothesis:
- - Matryoshka sub-models are within 1-3% of separately trained models
- - Matryoshka beats distillation (no teacher-student mismatch)
- - Single checkpoint, multiple operating points — simpler deployment
-
-What this shows:
- - Matryoshka is a practical elastic inference solution
- - The accuracy-efficiency tradeoff is smooth (no cliff)
- - One model replaces N separately trained models
-
-
-### Exp 4: Optimizer Comparison [STRONG]
-
-Setup:
- - Muon+Adam (our split): Muon for kernels, Adam for rest
- - Adam-only: AdamW with same schedule for all params
- - SGD+momentum: baseline optimizer
- - Same architecture, same data, same total steps
-
-Metrics:
- - Training loss curves (convergence speed)
- - Final val_ppl and call_f1
- - Wall-clock time per step
-
-Hypothesis:
- - Muon+Adam converges fastest (fewer steps to same loss)
- - Adam-only reaches similar final quality but slower
- - SGD is significantly worse (no adaptive rates for heterogeneous params)
- - Muon adds negligible wall-clock overhead (Newton-Schulz is 5 matmuls)
-
-What this shows:
- - Muon accelerates convergence for attention/FFN matrices
- - The split optimizer is justified: different params need different treatment
-
-
-### Exp 5: QAT + Pruning Compression Analysis [STRONG]
-
-Setup:
- - Evaluate combinations:
- Full precision, no pruning (baseline)
- INT4 quantized (post-training, no QAT)
- INT4 quantized (with QAT)
- 50% pruned (no QAT)
- INT4 + 50% pruned (both, with QAT)
- - Report model size, inference speed, and accuracy
-
-Metrics:
- | Condition | Size(MB) | tok/s | val_ppl | call_f1 |
- |----------------------|----------|-------|---------|---------|
- | FP16 baseline | | | | |
- | INT4 (no QAT) | | | | |
- | INT4 (with QAT) | | | | |
- | 50% pruned | | | | |
- | INT4 + 50% pruned | | | | |
-
- Also: group_size sweep for QAT {16, 32, 64, 128}
-
-Hypothesis:
- - QAT recovers most of the INT4 quantization loss (quant_ppl gap < 0.5)
- - Without QAT, INT4 has significant degradation at small model sizes
- - Pruning + QAT compound: 8x size reduction with <5% accuracy loss
- - Group size 32 is sweet spot (matches hardware vector width)
-
-What this shows:
- - QAT and pruning are essential for edge deployment
- - Co-designed training (both at group_size=32) enables extreme compression
- - Small models benefit more from QAT than large models (fewer redundant params)
-
-
-### Exp 6: Activation Comparison (DReLU vs SwiGLU vs GeGLU) [STRONG]
-
-Setup:
- - Three activations, everything else identical
- - Report: accuracy, activation sparsity, inference speed
-
-Metrics:
- | Activation | val_ppl | call_f1 | sparsity% | tok/s | quant_ppl |
-
-Hypothesis:
- - DReLU has highest activation sparsity (~65%)
- - SwiGLU has best raw val_ppl (smoothest gradients)
- - DReLU wins on call_f1 (sharper features for structured output)
- - DReLU wins on tok/s (no sigmoid/erf, sparse activations)
- - DReLU has best quant_ppl (sparse activations → better quantization)
-
-What this shows:
- - DReLU is the right choice for tool calling + edge deployment
- - The sparsity-accuracy tradeoff favors DReLU in this domain
-
-
-### Exp 7: Structured Loss Weight Sensitivity [STRONG]
-
-Setup:
- - Vary name weight: {1.0, 2.0, 3.0, 5.0, 10.0}
- - Vary value weight: {1.0, 2.0, 3.0, 5.0}
- - Key weight fixed at 1.5x
-
-Metrics (heatmap: name_weight x value_weight):
- - name_f1
- - call_f1
- - value_acc
-
-Hypothesis:
- - name_weight=1 (uniform): worst name_f1
- - name_weight=3: sweet spot (sufficient signal without overwhelming)
- - name_weight=10: name_f1 saturates, value_acc degrades (gradient
- starved on value tokens)
- - value_weight=2: good balance
-
-What this shows:
- - Structured weighting matters for structured output
- - The weights should reflect the error impact hierarchy:
- wrong name (catastrophic) > wrong value (incorrect) > wrong key (hallucinated)
-
-
-### Exp 8: Training Curriculum Effect [NICE-TO-HAVE]
-
-Setup:
- - With curriculum: sort by tool count (easy → hard within epoch)
- - Without curriculum: random shuffle
- - Same total data, same epochs
-
-Metrics:
- - val_ppl convergence curve
- - Per-tool-count call_f1 at epoch 1, 5, 10
-
-Hypothesis:
- - Curriculum helps early training (faster initial convergence)
- - By end of training, gap narrows
- - Biggest benefit on high-tool-count examples (model learns
- simple patterns first, then extends to complex cases)
-
-What this shows:
- - Curriculum is useful but not critical
- - Justifies the easy→hard ordering for tool-call training
-
-
-### Exp 9: ZCRMSNorm vs Standard RMSNorm [NICE-TO-HAVE]
-
-Setup:
- - ZCRMSNorm (γ init to 0, applied as (1+γ)*x/RMS)
- - Standard RMSNorm (γ init to 1, applied as γ*x/RMS)
- - Same architecture, same recipe
-
-Metrics:
- - Training loss curves (first 1000 steps: stability)
- - Final val_ppl and call_f1
- - Gradient norm trajectory
-
-Hypothesis:
- - ZCRMSNorm has smoother early training (no initial activation spikes)
- - Both converge to similar final quality
- - ZCRMSNorm shows lower gradient norm variance in first 500 steps
-
-What this shows:
- - Zero-centering helps training stability, especially at small scale
- - Free improvement: no added cost, just different initialization
-
-
-### Priority
-
-Must-have (core claims):
- 1. Enc-dec vs dec-only (Exp 1) — justifies architecture choice
- 2. Full recipe ablation (Exp 2) — proves co-design
- 3. Matryoshka Pareto (Exp 3) — proves elastic inference
-
-Strong additions:
- 4. Optimizer comparison (Exp 4) — justifies Muon+Adam
- 5. QAT + pruning compression (Exp 5) — deployment story
- 6. Activation comparison (Exp 6) — justifies DReLU
- 7. Loss weight sensitivity (Exp 7) — justifies structured weighting
-
-Nice-to-have:
- 8. Curriculum effect (Exp 8) — training efficiency
- 9. ZCRMSNorm ablation (Exp 9) — initialization choice
+ - We do not need MLPs for tiny models
+ - Structured loss weighting targets the actual error distribution
+ - QAT provides regularisation eliminating the need for dropout
+ - Attention is a nonlinearity in the absence of MLP
+
+----
+
+Our blocks (with no_feedforward=True) are essentially:
+
+ # encoder:
+ x = x + Attention(Norm(x))
+
+ # decoder:
+ x = x + SelfAttn(Norm(x))
+ x = x + CrossAttn(Norm(x))
+
+The residual connection is the only pathway preserving information between layers.
+Without FFN, there's no nonlinear feature transformation, each layer just incrementally mixes information via attention and adds it back.
+
+The argument for Kimi-style residual bypass here would be: if you remove the residual, each attention layer gets to fully rewrite the representation rather than just adding a delta.
+In a standard transformer with FFN, the FFN does the heavy "rewriting" work and the attention residual is fine as an additive refinement.
+But in our attention-only setup, forcing attention into an additive role may be limiting, it can only nudge the representation, never reshape it.
+Without FFN and without residuals, we lose the gradient highway entirely. Deep networks would struggle to train.
+Kimi's technique likely works in models that still have FFN layers (or some other nonlinearity) providing an alternate signal path.
+
+With the gated residual, each line becomes:
+
+ # gated residual — each sublayer gate is a learnable scalar, initialized to 0
+ # sigmoid(0) = 0.5, so training starts as half-strength standard residual
+ # gate → 1 recovers the standard residual exactly
+
+ # encoder (per layer)
+ g = sigmoid(gate)
+ x = x + g * Attention(Norm(x))
+
+ # decoder (per layer)
+ g_s = sigmoid(self_attn_gate)
+ g_c = sigmoid(cross_attn_gate)
+
+ x = x + g_s * SelfAttn(Norm(x))
+ x = x + g_c * CrossAttn(Norm(x))
+
+The full residual is always preserved. The gate only scales the sublayer contribution.
+At init (gate=0, g=0.5): half-strength sublayer addition.
+gate → 1 (g=1): standard residual. gate → -inf (g=0): identity, sublayer ignored.
\ No newline at end of file
diff --git a/notes/tool-selection.txt b/notes/tool-selection.txt
index f00f5dc..2b7dc3b 100644
--- a/notes/tool-selection.txt
+++ b/notes/tool-selection.txt
@@ -15,12 +15,17 @@ select the right tool(s) for a given query. Two challenges:
search_flights vs search_hotels vs search_restaurants). This is a
retrieval problem, not a generation problem.
+ (C) The model generates JSON tool calls autoregressively. While it learns
+ JSON structure well (99-100% parse rate), it can still hallucinate
+ invalid function names or argument keys. The valid tool definitions are
+ already in the encoder input — we can constrain the decoder to only
+ produce valid names/keys at inference time.
+
The standard approach is a separate retrieval model (e.g., embedding API)
to pre-filter tools before passing them to the generator. This adds a
second model, a second inference pass, and a mismatch between the
representations used for retrieval and generation.
-
## 2. Key Insight: Shared Encoder, Dual Use
Our encoder already processes both queries and tool definitions during
diff --git a/notes/tries.txt b/notes/tries.txt
deleted file mode 100644
index 7dfed76..0000000
--- a/notes/tries.txt
+++ /dev/null
@@ -1,267 +0,0 @@
-Selective Trie-Constrained Decoding for Tool-Call Generation
-=============================================================
-
-## 1. Problem
-
-The model generates JSON tool calls autoregressively. While it learns
-JSON structure well (99-100% parse rate), it can still hallucinate
-invalid function names or argument keys. The valid tool definitions are
-already in the encoder input — we can constrain the decoder to only
-produce valid names/keys at inference time.
-
-
-## 2. Key Insight: Constrain What's Broken, Not What Works
-
-The remaining errors are semantic, not syntactic:
-
- Error type Constrainable? Frequency
- ───────────────────────── ────────────── ─────────
- Wrong/hallucinated name YES (trie) Common
- Hallucinated param key YES (trie) Common
- Wrong argument values NO (open-ended) Common
- Missing arguments NO Moderate
- Malformed JSON NO (already 99%) Rare
-
-Full grammar approaches (Outlines, llama.cpp GBNF, guidance) enforce
-valid JSON on every token. This adds 10-30% overhead to solve a problem
-that doesn't exist. Our approach only constrains the ~5% of tokens that
-are inside name/key spans.
-
-
-## 3. Architecture
-
-Three components: a character-level trie, a JSON state machine, and
-a token masking layer.
-
-### 3a. Trie
-
-Given tools: get_weather(location, unit), get_stock(symbol)
-
-Name trie: Param trie (get_weather):
-
- (root) (root)
- | / \
- g l u
- | | |
- e o n
- | | |
- t c i
- | | |
- _ a t*
- / \ |
- w s t
- | | |
- e t i
- | | |
- a o o
- | | |
- t c n*
- | |
- h k* (* = terminal node)
- |
- e
- |
- r*
-
-One name trie shared across all tools. One param trie per function,
-built from its parameter keys.
-
-### 3b. JSON State Machine
-
-Tracks position in output to know when to constrain:
-
- ┌──────┐ "name":" ┌─────────┐ closing " ┌──────┐
- │ FREE │────────────→│ IN_NAME │────────────→│ FREE │
- └──────┘ └─────────┘ └──────┘
- │ │
- │ "arguments":{ then {" or ," │
- │ ┌────────────┐ closing " │
- └───────────────→│ IN_ARG_KEY │─────────────→┘
- └────────────┘
-
-State transitions driven by suffix matching on generated text.
-Tracks nesting depth to avoid false triggers inside nested values.
-Tracks string values with escape handling to avoid false triggers
-inside string content.
-
-### 3c. Token Masking
-
-At each constrained step:
-
- 1. Look up current trie node (root + constrained_buf prefix)
- 2. Gather valid first chars from trie children (+ closing " if terminal)
- 3. For each candidate token (via TokenIndex first-char lookup):
- - Walk trie char-by-char through token text
- - If token contains ", check trie node is terminal at that point
- - Mark as valid or invalid
- 4. Set invalid token logits to -inf
- 5. Fallback: if ALL tokens masked, revert to unconstrained + log warning
-
-Example (IN_NAME state, after consuming "get_"):
-
- Trie node at "get_" has children: {w, s}
- Scan TokenIndex for tokens starting with 'w' or 's':
- "weather" → walk trie: w→e→a→t→h→e→r, all valid → ALLOW
- "we" → walk trie: w→e, valid prefix → ALLOW
- "stock" → walk trie: s→t→o→c→k, terminal → ALLOW
- "sun" → walk trie: s→u, 'u' not child of 's' → BLOCK
-
-Multi-char token boundary handling:
- Token "ther\"" (closing quote mid-token):
- Walk trie for 't','h','e','r' → valid path
- Hit '"' → check if node is terminal → yes (weather*) → ALLOW
- Correctly handles BPE tokens that span the name boundary.
-
-
-## 4. Feedback Effect
-
-The corrected token feeds back into the decoder buffer. All subsequent
-generation conditions on the valid name:
-
- Without constraint: ..."name":"get_weathr","arguments":{"locaton":...
- ~~~~~~~~~~~~ wrong ~~~~~ ~~~~~~~~ wrong ~~~~
-
- With constraint: ..."name":"get_weather","arguments":{"location":...
- ~~~~~~~~~~~~ forced ~~~~ ~~~~~~~~~ model sees
- correct name, generates
- better arguments
-
-This is why in-place correction beats post-processing — the decoder
-sees correct context when generating arguments.
-
-
-## 5. Comparison with Full Grammar Approaches
-
- ┌─────────────────────────────────────────────────────────┐
- │ Full grammar (CFG/PDA) │
- │ │
- │ Constrains EVERY token: brackets, colons, commas, │
- │ string delimiters, value types, nesting, whitespace. │
- │ Guarantees syntactically valid JSON always. │
- │ │
- │ Cost: FSM/PDA transition on every token, large state │
- │ space, precomputed token masks per state, high memory. │
- │ Typical overhead: 10-30% slowdown on all tokens. │
- └─────────────────────────────────────────────────────────┘
-
- ┌─────────────────────────────────────────────────────────┐
- │ Selective constraint (trie + state machine) │
- │ │
- │ Only constrains tokens inside "name":"___" and │
- │ argument key "___" spans. Everything else │
- │ (structure, values, punctuation) is unconstrained. │
- │ │
- │ Cost: lightweight state machine tracks JSON position │
- │ via suffix matching. Trie walk + token masking only │
- │ activates during name/key spans (~5% of tokens). │
- │ Zero overhead on remaining ~95% of tokens. │
- └─────────────────────────────────────────────────────────┘
-
-Why full grammar is overkill here:
- - 99-100% JSON parse rate means structure is already learned
- - Full grammar pays per-token cost for ~0% parse improvement
- - Can force the model into corners where the most likely valid
- continuation is nonsensical (distorts value generation)
-
-Why selective works:
- - Targets the actual failure mode (hallucinated names/keys)
- - Near-zero cost (only active on ~5% of tokens)
- - Never distorts value generation
- - Corrected tokens improve downstream generation via feedback
-
-
-## 6. Implementation Details
-
- token_strings / TokenIndex: cached per tokenizer (built once)
- Device-to-host transfer: only when state machine is active
- (IN_NAME or IN_ARG_KEY); FREE state stays on jnp.argmax
- Enabled by default; disable with --no-constrained
-
-
-## 7. Experiment Design
-
-All trie experiments use our val set (5000 samples) — clean controlled
-variable where only the decoding mode changes. Exact reference calls
-give precise name_f1, param_halluc, value_acc. Per-tool-count breakdown
-already implemented. Reproducible with a seed.
-
-### Exp 1: Ablation — Constrained vs Unconstrained [MUST-HAVE]
-
-Same checkpoint, same val set, three conditions:
- (a) Unconstrained (--no-constrained)
- (b) Name-only constraint (trie on function names, keys unconstrained)
- (c) Full selective constraint (trie on names + argument keys)
-
- | Condition | name_f1 | call_f1 | exact_match | param_halluc | param_miss | value_acc | json_parse |
-
-Expects: name_f1 improves (a→b), param_halluc drops (b→c), json_parse
-unchanged across all three (confirms structure isn't the problem).
-Needs a "names_only" mode flag in ConstrainedDecoder.
-
-### Exp 2: Latency Overhead [MUST-HAVE]
-
-500 samples, wall-clock time and tok/s, three conditions, 3 runs each.
-
- | Condition | tok/s | Overhead |
-
-Also report: % tokens where constraint was active, avg time in
-apply_constraints per constrained token.
-Expects: near-zero overhead because constraints activate on ~5% of tokens.
-
-### Exp 3: Error Analysis — What Constraints Fix [MUST-HAVE]
-
-Run constrained and unconstrained on same 1000 samples. Categorize:
- (a) Both correct → constraint irrelevant
- (b) Unconstrained wrong, constrained correct → constraint helped
- (c) Both wrong (same error) → constraint irrelevant
- (d) Both wrong (different error) → constraint changed but didn't fix
- (e) Unconstrained correct, constrained wrong → constraint hurt (should be ~0)
-
-For category (b), sub-classify:
- - Misspelled function name (get_weathr → get_weather)
- - Hallucinated function name (fetch_weather → get_weather)
- - Hallucinated parameter key (city → location)
- - Misspelled parameter key (locaton → location)
-
-Present as sankey diagram + table of top-10 fixes with before/after.
-Category (e) should be empty — proves constraints are safe.
-
-### Exp 4: Breakdown by Tool Count [STRONG]
-
-Use existing per_count breakdown (1-10 tools per example).
-Grouped bar chart: name_f1 and param_halluc per bucket, constrained vs not.
-
-Expects: more tools → more names to confuse → bigger constraint benefit.
-Shows constraint value scales with schema complexity.
-
-### Exp 5: Downstream Effect — Name Correction Improves Arguments [STRONG]
-
-Focus on examples where unconstrained gets the name wrong.
-Compare value_acc and args_acc when name is wrong vs corrected.
-
-Expects: corrected name → better arguments (feedback effect).
-Justifies in-place correction over post-processing.
-
-### Exp 6: Benefit Across Training Epochs [STRONG]
-
-Checkpoints at epochs 1, 3, 5, 7, 10. Each evaluated with and without.
-Line plot: call_f1 constrained vs unconstrained across epochs.
-
-Expects: biggest delta mid-training ("almost right" phase), positive
-everywhere. Shows constraints are complementary to training.
-
-### Exp 7: Full Grammar Comparison [NICE-TO-HAVE]
-
-Implement Outlines-style full JSON grammar constraint as baseline.
-Three-way comparison: unconstrained, full grammar, selective trie.
-
-Expects: full grammar has ~same call_f1 but much higher overhead.
-Strongest competitive claim but most implementation work.
-
-### Exp 8: Out-of-Distribution Tool Schemas [NICE-TO-HAVE]
-
-Synthetic tools with names/keys not in training data (20-50 examples).
-Constrained vs unconstrained.
-
-Expects: OOD names have highest hallucination rate → biggest delta.
-Shows value for custom/long-tail APIs. Needs manual data creation.
diff --git a/notes/voice-to-action.txt b/notes/voice-to-action.txt
new file mode 100644
index 0000000..e69de29
diff --git a/src/cli.py b/src/cli.py
index 1b4a018..91a55e4 100644
--- a/src/cli.py
+++ b/src/cli.py
@@ -15,7 +15,7 @@ def main():
p = sub.add_parser("train", add_help=False)
p.add_argument("--checkpoint", type=str, default=None)
- p.add_argument("--epochs", type=int, default=3)
+ p.add_argument("--epochs", type=int, default=1)
p.add_argument("--batch-size", type=int, default=32)
p.add_argument("--lr", type=float, default=3e-4)
p.add_argument("--muon-lr", type=float, default=0.02)
@@ -46,7 +46,7 @@ def main():
p.add_argument("--activation", type=str, default="swiglu", choices=["drelu", "swiglu", "geglu"])
p.add_argument("--num-memory-slots", type=int, default=128,
help="(DEPRECATED — ignored, kept for checkpoint compat)")
- p.add_argument("--mat-factors", type=int, nargs="*", default=[2, 4, 8],
+ p.add_argument("--mat-factors", type=int, nargs="*", default=[2, 4],
help="Matryoshka FFN shrink factors, e.g. 2=half width (default: 2 4)")
p.add_argument("--dropout", type=float, default=0.0,
help="Dropout rate for residual connections (default: 0.0)")
@@ -58,6 +58,8 @@ def main():
help="Dimension of contrastive projection head (default: 128)")
p.add_argument("--conv-kernel-size", type=int, default=0,
help="Conformer conv kernel size in encoder (0=disabled, 15=default when enabled)")
+ p.add_argument("--no-feedforward", action=argparse.BooleanOptionalAction, default=True,
+ help="Remove feedforward layers entirely (default: True)")
p = sub.add_parser("tokenize", add_help=False)
p.add_argument("--max-samples", type=int, default=None,
diff --git a/src/data.py b/src/data.py
index 48d114c..d09a363 100644
--- a/src/data.py
+++ b/src/data.py
@@ -380,8 +380,8 @@ def load_tool_calls(split="train", max_samples=None, return_global_indices=False
perm = rng.permutation(n)
val_size = min(5000, int(n * 0.1))
- val_indices = perm[:val_size]
- train_indices = perm[val_size:]
+ val_indices = perm[-val_size:]
+ train_indices = perm[:-val_size]
if split in ("validation", "val", "test"):
global_indices = val_indices.astype(np.int64)
diff --git a/src/eval.py b/src/eval.py
index 9319a48..c9aeb79 100644
--- a/src/eval.py
+++ b/src/eval.py
@@ -345,7 +345,7 @@ def benchmark_tool_calls(model, params, tokenizer, num_samples=200, max_gen_len=
}
-def benchmark_retrieval(model, params, tokenizer, num_samples=500, max_len=256, ks=(1, 3, 5, 10), ds=None):
+def benchmark_retrieval(model, params, tokenizer, num_samples=500, max_len=256, ks=(1, 2, 3, 4, 5), ds=None):
"""Benchmark contrastive retrieval: Recall@k and MRR over validation set.
For each query, ranks all tools from that example by cosine similarity
diff --git a/src/model.py b/src/model.py
index df3f547..9e34ea3 100644
--- a/src/model.py
+++ b/src/model.py
@@ -50,6 +50,7 @@ class TransformerConfig:
contrastive_dim: int = 128
conv_kernel_size: int = 0
enable_speech: bool = False
+ no_feedforward: bool = True
@property
def jax_dtype(self):
@@ -221,7 +222,7 @@ class ConvModule(nn.Module):
class EncoderBlock(nn.Module):
- """Pre-norm self-attention + pre-norm FFN with residual connections."""
+ """Pre-norm self-attention + pre-norm FFN with gated residual connections."""
num_heads: int
num_kv_heads: int
d_model: int
@@ -231,27 +232,32 @@ class EncoderBlock(nn.Module):
activation: str = "drelu"
dropout_rate: float = 0.0
conv_kernel_size: int = 0
+ no_feedforward: bool = True
@nn.compact
def __call__(self, x, mask=None, rope=None, ffn_mask=None, deterministic=True):
+ gate = nn.sigmoid(self.param("attn_gate", jinit.zeros, ())).astype(self.dtype)
residual = x
x = ZCRMSNorm(dtype=self.dtype)(x)
x = MultiHeadAttention(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers, self.dtype, name="self_attn")(
x, x, mask=mask, rope=rope
)
- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual
+ x = residual + gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)
if self.conv_kernel_size > 0:
+ conv_gate = nn.sigmoid(self.param("conv_gate", jinit.zeros, ())).astype(self.dtype)
residual = x
x = ZCRMSNorm(dtype=self.dtype)(x)
x = ConvModule(self.d_model, self.conv_kernel_size, self.num_layers,
self.dtype, name="conv_module")(x)
- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual
+ x = residual + conv_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)
- residual = x
- x = ZCRMSNorm(dtype=self.dtype)(x)
- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)
- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual
+ if not self.no_feedforward:
+ ffn_gate = nn.sigmoid(self.param("ffn_gate", jinit.zeros, ())).astype(self.dtype)
+ residual = x
+ x = ZCRMSNorm(dtype=self.dtype)(x)
+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)
+ x = residual + ffn_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)
return x
@@ -267,6 +273,7 @@ class _EncoderScanBody(nn.Module):
activation: str = "drelu"
dropout_rate: float = 0.0
conv_kernel_size: int = 0
+ no_feedforward: bool = True
deterministic: bool = True
@nn.compact
@@ -275,7 +282,7 @@ class _EncoderScanBody(nn.Module):
x = EncoderBlock(
self.num_heads, self.num_kv_heads, self.d_model, self.d_ff,
self.num_layers, self.dtype, self.activation, self.dropout_rate,
- self.conv_kernel_size,
+ self.conv_kernel_size, self.no_feedforward,
)(x, mask, rope, ffn_mask, self.deterministic)
return (x, mask, rope, ffn_mask), None
@@ -299,7 +306,7 @@ class Encoder(nn.Module):
(x, _, _, _), _ = ScanBlock(
cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff,
cfg.total_layers, dt, cfg.activation, cfg.dropout_rate,
- cfg.conv_kernel_size, deterministic, name="layers",
+ cfg.conv_kernel_size, cfg.no_feedforward, deterministic, name="layers",
)((x, mask, rope, ffn_mask), None)
x = ZCRMSNorm(dtype=dt, name="final_norm")(x)
@@ -315,27 +322,32 @@ class DecoderBlock(nn.Module):
dtype: jnp.dtype = jnp.bfloat16
activation: str = "drelu"
dropout_rate: float = 0.0
+ no_feedforward: bool = True
@nn.compact
def __call__(self, x, encoder_out, self_mask=None, cross_mask=None, rope=None, ffn_mask=None, deterministic=True):
+ self_gate = nn.sigmoid(self.param("self_attn_gate", jinit.zeros, ())).astype(self.dtype)
residual = x
x = ZCRMSNorm(dtype=self.dtype)(x)
x = MultiHeadAttention(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers, self.dtype, name="self_attn")(
x, x, mask=self_mask, rope=rope
)
- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual
+ x = residual + self_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)
+ cross_gate = nn.sigmoid(self.param("cross_attn_gate", jinit.zeros, ())).astype(self.dtype)
residual = x
x = ZCRMSNorm(dtype=self.dtype)(x)
x = MultiHeadAttention(self.num_heads, self.num_kv_heads, self.d_model, self.num_layers, self.dtype, name="cross_attn")(
x, encoder_out, mask=cross_mask
)
- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual
+ x = residual + cross_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)
- residual = x
- x = ZCRMSNorm(dtype=self.dtype)(x)
- x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)
- x = nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic) + residual
+ if not self.no_feedforward:
+ ffn_gate = nn.sigmoid(self.param("ffn_gate", jinit.zeros, ())).astype(self.dtype)
+ residual = x
+ x = ZCRMSNorm(dtype=self.dtype)(x)
+ x = FeedForward(self.d_model, self.d_ff, self.num_layers, self.dtype, self.activation)(x, ffn_mask=ffn_mask)
+ x = residual + ffn_gate * nn.Dropout(rate=self.dropout_rate)(x, deterministic=deterministic)
return x
@@ -351,6 +363,7 @@ class _DecoderScanBody(nn.Module):
dtype: jnp.dtype = jnp.bfloat16
activation: str = "drelu"
dropout_rate: float = 0.0
+ no_feedforward: bool = True
deterministic: bool = True
@nn.compact
@@ -359,6 +372,7 @@ class _DecoderScanBody(nn.Module):
x = DecoderBlock(
self.num_heads, self.num_kv_heads, self.d_model, self.d_ff,
self.num_layers, self.dtype, self.activation, self.dropout_rate,
+ self.no_feedforward,
)(x, encoder_out, self_mask, cross_mask, rope, ffn_mask, self.deterministic)
return (x, encoder_out, self_mask, cross_mask, rope, ffn_mask), None
@@ -381,7 +395,7 @@ class Decoder(nn.Module):
(x, _, _, _, _, _), _ = ScanBlock(
cfg.num_heads, cfg.num_kv_heads, cfg.d_model, cfg.d_ff,
cfg.total_layers, dt, cfg.activation, cfg.dropout_rate,
- deterministic, name="layers",
+ cfg.no_feedforward, deterministic, name="layers",
)((x, encoder_out, self_mask, cross_mask, rope, ffn_mask), None)
x = ZCRMSNorm(dtype=dt)(x)
diff --git a/src/train.py b/src/train.py
index 8f8c24b..dcd861d 100644
--- a/src/train.py
+++ b/src/train.py
@@ -531,6 +531,7 @@ def train(args):
contrastive_dim=getattr(args, "contrastive_dim", 128),
conv_kernel_size=getattr(args, "conv_kernel_size", 0),
enable_speech=getattr(args, "enable_speech", False),
+ no_feedforward=getattr(args, "no_feedforward", True),
)
global _GROUP_SIZE, _MAT_FACTORS, _MAT_FF_WIDTHS, _D_FF, _CONTRASTIVE_WEIGHT
@@ -538,7 +539,7 @@ def train(args):
_CONTRASTIVE_WEIGHT = getattr(args, "contrastive_weight", 0.1)
_D_FF = config.d_ff
mat_factors_raw = getattr(args, "mat_factors", None)
- if mat_factors_raw:
+ if mat_factors_raw and not config.no_feedforward:
_MAT_FACTORS = tuple(f for f in mat_factors_raw if f > 1)
_MAT_FF_WIDTHS = tuple(config.d_ff // f for f in _MAT_FACTORS)
else:
@@ -971,6 +972,9 @@ def train(args):
_gen_model = eval_model
_gen_label = "full"
+ _mat_params = _quantize_params(_mat_params, group_size=_GROUP_SIZE)
+ _gen_label += " INT4"
+
all_preds = []
_gen_t0 = time.perf_counter()
for _ei in tqdm(range(0, len(all_eval_examples), _EVAL_BATCH),