Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
cleanup
HenryNdubuakuddf80e085239
diff --git a/docs/matryoshka_results.md b/docs/matryoshka_results.mddeleted file mode 100644index 0b22635..0000000--- a/docs/matryoshka_results.md+++ /dev/null@@ -1,123 +0,0 @@-# Matryoshka — Results--## Approach--Heterogeneous batch matryoshka: a per-batch-item prefix mask zeros out FFN neurons beyond the target width. Single forward-backward pass, single optimizer step. All widths trained simultaneously.--- **Full width**: d_ff = 2048 (all neurons active)-- **2x**: d_ff = 1024 (first 1024 neurons)-- **4x**: d_ff = 512 (first 512 neurons)-- **8x**: d_ff = 256 (first 256 neurons)--d_model, attention heads, and embeddings are unchanged across all widths.--Two masking scopes:-- **FFN-only**: masks decoder FFN and encoder local FFN. MLP-Mixer token/channel mixing layers are NOT masked.-- **Full matryoshka** (current): masks ALL FFN layers including MLP-Mixer token/channel mixing.--Two input modes:-- **Unique input** (default): each batch item is a unique sample, assigned a fixed width based on its position in the batch. Full data diversity per epoch.-- **Shared input** (`--mat-shared-input`): each unique sample is repeated across all widths. Smaller unique batch but every sample trains every width. Epoch = 4x more steps to see all data.--## Training Config--All runs use the same architecture and hyperparameters:--| Setting | Value |-|---|---|-| d_model | 512 |-| d_ff | 2048 |-| Heads (KV) | 8 (8) |-| Layers | 8 enc / 4 dec |-| Memory slots | 64 |-| Activation | dReLU |-| Dtype | bfloat16 |-| Devices | 8 (TPU v6e-8) |-| Batch | 32 x 8 = 256 |-| LR (Adam) | 0.0003 x 8 = 0.0024 |-| LR (Muon) | 0.02 -> 0.0566 |-| Sparsity | 50% target (gradual cubic, group=32) |-| Speech | every 3 text steps |--## Results (1 epoch, val PPL)--| Model | No mat (main) | FFN-only | FFN-only shared | **Full mat (FFN + mixer)** |-|---|---|---|---|---|-| Full (d_ff=2048) | **4.52** | 4.66 | 5.11 | 4.82 |-| 2x (d_ff=1024) | - | 4.66 | 5.11 | 4.82 |-| 4x (d_ff=512) | - | 4.68 | 5.11 | 4.85 |-| 8x (d_ff=256) | - | **4.80** | 5.20 | 4.97 |-| Quant (INT4 g32) | **4.59** | 4.75 | 5.20 | 4.94 |--> No-mat sub-model PPLs omitted — output-only slice does not constrain internal computation, so sub-model PPLs are misleading.--| Metric | No mat | FFN-only | FFN-only shared | Full mat |-|---|---|---|---|---|-| Masking scope | Output-only slice | Decoder+encoder FFN | Decoder+encoder FFN | All FFN + mixer |-| Parameters | 87,666,176 | 87,666,176 | 87,666,176 | 88,535,040 |-| Total steps | 11,038 | 11,038 | 44,156 | 11,038 |-| Unique items/batch | 32/dev | 32/dev | 8/dev x 4 | 32/dev |-| Speed | ~19.3 it/s | 11.8 it/s | 20.9 it/s | 10.2 it/s |-| Wall time | ~10 min | ~16 min | ~36 min | ~18 min |-| Avg train loss | 4.43 | 4.42 | 3.12 | - |-| Final train loss | 1.51 | 1.53 | 1.66 | - |-| Text val PPL | **4.52** | 4.66 | 5.11 | 4.82 |-| Quant val PPL | **4.59** | 4.75 | 5.20 | 4.94 |-| Weight sparsity | 50.02% | 50.01% | 68.28% | 49.52% |-| Speech val PPL | 87.26 | 92.59 | 78.64 | 83.56 |-| Speech WER | 1.0000 | 0.9695 | 0.9618 | 0.9924 |-| Throughput | 135.7 tok/s | 46.3 tok/s | 156.8 tok/s | 146.6 tok/s |--## Analysis--**No-mat full-model PPL (4.52) is the best**, as expected — it doesn't spend any capacity on sub-model training.--**FFN-only (4.66) is the best matryoshka approach** for text PPL — it pays +0.14 over no-mat for genuinely training 4 widths simultaneously, with only a +0.14 gap from full to 8x (4.66→4.80).--**Full matryoshka (4.82) pays an additional +0.16 over FFN-only** for also masking the mixer layers. The sub-model spread is similarly tight (full→8x: 0.15 PPL). The extra cost buys truly independent sub-models where even the mixer's token and channel mixing operate at reduced width.--**Shared input is not worth it** (4.66 vs 5.11 for FFN-only). Unique input with random width assignment per item is sufficient — every sample doesn't need to train every width.--**Full mat is slower** (10.2 it/s vs 11.8 FFN-only) because the mixer FFNs are also heterogeneously masked, adding compute overhead.--### Recommendation--Use **unique input** (default, no flag) for best quality. Choose between FFN-only and full matryoshka based on whether mixer-level sub-models are needed:-- **FFN-only**: better text PPL (4.66 vs 4.82), faster (11.8 vs 10.2 it/s). Sub-models share full mixer capacity.-- **Full matryoshka**: truly independent sub-models at every layer. Slightly worse but sub-models are more self-contained for export.--## Training Speed--| Approach | it/s | Notes |-|---|---|---|-| No mat (output-only slice) | ~19.3 | 1 forward + logit slicing (no true sub-models) |-| FFN-only unique input | 11.8 | Single forward, heterogeneous batch (decoder+encoder FFN) |-| FFN-only shared input | 20.9 | Single forward, repeated inputs (XLA optimizes) |-| Full mat unique input | 10.2 | Single forward, heterogeneous batch (all FFN + mixer) |--## Ablation: Shared Input at 4x Batch Size (FFN-only)--**Question**: Does training every sample at all widths help, if we control for gradient quality?--With `--mat-shared-input --batch-size 128`, each step has 32 unique samples repeated 4x (one per width) — same unique samples per step as the default unique-input run at batch=32. The only difference is that every sample trains every width, at the cost of 4x compute per step.--| Model | FFN unique (bs=32) | FFN shared (bs=128) |-|---|---|---|-| Full (d_ff=2048) | **4.66** | 4.67 |-| 2x (d_ff=1024) | **4.66** | 4.67 |-| 4x (d_ff=512) | **4.68** | 4.69 |-| 8x (d_ff=256) | **4.80** | 4.81 |-| Quant (INT4 g32) | **4.75** | 4.77 |-| Speed | 11.8 it/s | 5.6 it/s |-| Wall time | ~16 min | ~35 min |-| Sparsity | 50.01% | 50.01% |--**Result**: Essentially identical PPL (4.66 vs 4.67). Repeating inputs across widths provides no benefit when gradient quality (unique samples per step) is controlled. Random width assignment per item is sufficient.--This confirms **unique input at batch=32 is optimal**: same quality, 2x faster.--## Notes--- All runs are 1-epoch on TinyStories + LibriSpeech. Multi-epoch runs would improve all metrics.-- Speech WER is poor across all methods — expected with interleaved training and limited speech data.-- The heterogeneous batch approach uses the same memory as a single full-width forward (~1x). No overhead from matryoshka.diff --git a/docs/posttraining.md b/docs/posttraining.mddeleted file mode 100644index 2d543e2..0000000--- a/docs/posttraining.md+++ /dev/null@@ -1,110 +0,0 @@-## Stage 0: Induction Head Pre-training (Tool Call Head Warm-up)--Before training the tool call head on real tool data, we pre-train it on synthetic structured retrieval tasks to teach the mechanical skill of deterministic symbol lookup and copying — the core operation of tool calling.--```mermaid-graph LR- %% Synthetic data- TL[Synthetic Tool List<br>random token IDs] --> Tool_dec[Tool Call Head<br>frozen base + LoRA]- CTX[Synthetic Context<br>with embedded values] --> Enc[Shared Encoder<br>frozen]- Enc --> Z[z-latent]- Z --> Tool_dec- Tool_dec --> Loss[Structured Retrieval Loss]-```--### Motivation--Two recent results motivate this stage:--1. **Formal language pre-pretraining transfers to natural language** ([Chiang et al., ACL 2025](https://aclanthology.org/2025.acl-long.478.pdf)): Pre-training a transformer on hierarchically structured formal languages (k-Shuffle Dyck) before natural language training provides positive transfer, particularly for *verbatim retrieval*. 500 steps of formal language pre-pretraining can replace ~4000 steps of natural language training (MRS ≈ 8). Crucially, the copy language $ww$ does *not* help — hierarchical structure with cross-serial dependencies is key.--2. **Random embeddings accelerate induction head formation** ([Liu, 2026](https://kindxiaoming.github.io/blog/2026/sparse-attention-8/)): Resampling token embeddings at each training step forces the model to learn symbolic (pattern-based) attention patterns rather than memorizing numeric correlations. This dramatically accelerates the formation of induction heads that implement the abstract rule $[A][B]\ldots[A] \rightarrow [B]$ regardless of token identity.--### Why this helps tool calling--Tool calling is fundamentally structured associative recall with hierarchical dependencies:-- **Level 1** (tool selection): Match intent in context to one of $K$ tools — analogous to bracket matching in k-Shuffle Dyck-- **Level 2** (argument filling): For each argument of the selected tool, locate and copy the correct value from context — cross-serial dependency retrieval--By pre-training on synthetic data with random embeddings, the head learns the *symbolic pattern* of retrieval (attend to the right key in the list, copy it exactly, fill the value) independently of what specific tokens represent. The latent-based conditioning (which tool to actually call given the audio/text input) is then learned naturally during real tool call training.--### Curriculum with decaying embedding noise--| Phase | Steps | Tools | Args | Embedding noise | Notes |-|-------|-------|-------|------|-----------------|-------|-| 0a | ~500 | 2 | 1 each | Fully random (resample each step) | Pure symbolic learning |-| 0b | ~500 | 5 | 2–3 | Fully random | Scale up retrieval complexity |-| 0c | ~1000 | 10 | 2–5 | Gaussian $\sigma$: 1.0 → 0.3 | Values in natural context, distractors |-| 0d | ~500 | Full | 2–5 | Gaussian $\sigma$: 0.1 | Transition to real embedding geometry |--At each step, tool/argument names are drawn from random token IDs (or the embedding matrix is resampled). A synthetic context is generated containing the values for arguments of one tool, embedded in distractor text. The target is the structured output selecting the correct tool and filling in the correct arguments. **Constrained decoding is used during pre-training** to match the final tool call generation setup.--### LoRA adapter strategy--The tool call head is initialized from the pretrained text reconstruction head. To preserve text generation capability while learning the retrieval circuit:-1. Freeze the base weights (text reconstruction head)-2. Attach LoRA adapters (rank $r$) to the attention projections-3. Train only the LoRA parameters on the synthetic retrieval task-4. After Stage 0: merge LoRA weights into the base weights, producing a head with both text generation and retrieval capabilities baked in-5. Proceed to real tool call training (Stage 2) with all weights unfrozen-------## Transcription Training-```mermaid-graph LR- %% Inputs- AT_pair[Audio-Text pair] --> A_in[Audio]- AT_pair --> T_in[Text]- A_in --> A_emb[Audio Embedding]- T_in --> T_emb[Text Embedding]-- %% Embeddings to Encoder- A_emb --> Enc[Shared Encoder]- T_emb --> Enc-- %% Encoder to Latents- Enc --> ZA[z-audio]- Enc --> ZT[z-text]-- %% Audio Transcription Branch- ZA --> Tr_dec[Transcription Head]- Tr_dec --> A_loss[Audio Transcription Loss]-- %% Contrastive Alignment Branch- ZA --> Proj[Audio Projector]- Proj --> C_loss[SigLip Loss]- ZT --> C_loss-```--## Audio-Text Tool call training-```mermaid-graph LR- %% Inputs- AT_pair[Audio-Text prompt pair] --> A_in[Audio Prompt]- AT_pair --> T_in[Text Prompt]- A_in --> A_emb[Audio Embedding]- T_in --> T_emb[Text Embedding]-- %% Embeddings to Encoder- T_emb --> Enc[Shared Encoder]- A_emb --> Enc-- %% Encoder to Latents- Enc --> ZA[z-audio]- Enc --> ZT[z-text]-- %% Contrastive Alignment Branch- ZA --> Proj[Audio Projector]- Proj --> C_loss[SigLip Loss]- ZT --> C_loss-- %% Tool Call Branch- ZA --> Tool_dec[Tool Call Head]- ZT --> Tool_dec- P_in[Tool List] --> Tool_dec- Tool_dec --> Tool_loss[Text Tool Call Loss]- Tool_dec --> ATool_loss[Audio Tool Call Loss]-```--Audio transcription training and tool call training can be done at the same time, while the Audio tool call training should be done separately at the end.\ No newline at end of filediff --git a/docs/pretraining.md b/docs/pretraining.mddeleted file mode 100644index 23a4fd0..0000000--- a/docs/pretraining.md+++ /dev/null@@ -1,67 +0,0 @@-### Stage 1 (Unlabeled): Recon + Adversarial Alignment--```mermaid-graph LR- %% Inputs- A_in[Audio] --> A_emb[Audio Embedding]- T_in[Text] --> T_emb[Text Embedding]-- %% Embeddings to Encoder- A_emb --> Enc[Shared Encoder]- T_emb --> Enc-- %% Encoder to Latents- Enc --> ZA[z-audio]- Enc --> ZT[z-text]-- %% Audio Reconstruction Branch- ZA --> A_dec[Audio Decoder]- A_dec --> A_loss[Audio Reconstruction Loss]-- %% Text Reconstruction Branch- ZT --> T_dec[Text Decoder]- T_dec --> T_loss[Text Reconstruction Loss]-- %% Adversarial Alignment Branch- ZA --> Proj[Audio Projector]- Proj --> Adv[Alignment Loss]- ZT --> Adv-```--Let $a \sim \mathcal{D}_a$ denote an audio sample and $x \sim \mathcal{D}_x$ denote a text sample (unpaired across datasets). A shared encoder $f_\theta(\cdot)$ maps either modality to a latent representation:-$$z_a = f_\theta(a), \qquad z_x = f_\theta(x).$$--We attach modality-specific reconstruction decoders:-$$\hat a = r_{\phi_a}(z_a), \qquad \hat x = r_{\phi_x}(z_x),$$-with reconstruction losses (chosen to match the underlying representation, e.g. $\ell_1/\ell_2$ for continuous targets or cross-entropy for discrete tokens):-$$\mathcal{L}_{\mathrm{rec\text{-}a}}(a) = \ell_a(\hat a, a), \qquad \mathcal{L}_{\mathrm{rec\text{-}x}}(x) = \ell_x(\hat x, x).$$--Taking expectations over the unlabeled datasets yields:-$$\mathcal{L}_{\mathrm{rec\text{-}a}} = \mathbb{E}_{a\sim \mathcal{D}_a}\!\left[\mathcal{L}_{\mathrm{rec\text{-}a}}(a)\right], \qquad \mathcal{L}_{\mathrm{rec\text{-}x}} = \mathbb{E}_{x\sim \mathcal{D}_x}\!\left[\mathcal{L}_{\mathrm{rec\text{-}x}}(x)\right].$$--To align the audio and text latent distributions while allowing audio-specific detail to be stripped, we introduce an audio projector $p_\psi(\cdot)$:-$$u_a = p_\psi(z_a).$$--We train a modality discriminator $d_\omega(\cdot)\in(0,1)$ to predict whether its input came from the audio pathway (label $1$ for $u_a$) or the text pathway (label $0$ for $z_x$). The discriminator is trained with binary cross-entropy:-$$\mathcal{L}_D(\omega) = \mathbb{E}_{a\sim \mathcal{D}_a}\!\left[-\log d_\omega(u_a)\right] + \mathbb{E}_{x\sim \mathcal{D}_x}\!\left[-\log\big(1-d_\omega(z_x)\big)\right].$$--Stage 1 training follows the standard minimax objective: reconstruction decoders (and the shared encoder) minimize reconstruction error, while the encoder/projector are trained adversarially to *fool* the discriminator. Let $\Theta=\{\theta,\phi_a,\phi_x,\psi\}$ denote parameters of the shared encoder, reconstruction decoders, and projector. We optimize:-$$\min_{\Theta}\;\max_{\omega}\;\; \mathcal{L}_{\mathrm{rec\text{-}a}} + \mathcal{L}_{\mathrm{rec\text{-}x}} \;-\;\lambda_{\mathrm{adv}}\,\mathcal{L}_D(\omega;\Theta),$$-where $\lambda_{\mathrm{adv}}\ge 0$ controls the strength of adversarial alignment.--## **Spec and Ablations**-1. **Audio Embedding**: There are essentially two options. First, we can take the raw waverform and and use convolutions on it Moonshine style to construct features. Second, we can generate the MEL bins and process them Whisper style. We can either do patches ViT style or again just conv over MEL Whisper style. The audio dataset we will be using is TBD (maybe granary), the audio will be prepared by augmenting it with slight natural noise, then we mask out different portions of the MEL bin and then the final prediction must be done to recreate the inital unmasked mel bins.--2. **Audio Decoder**: The audio decoder that we will be using has to reconstruct the MEL bins. We will probably make everything fixed size, padded to 30s. The decoder is going to be a shallow non-causal transformer with learned positional embeddings as the initial query. The decoder has to be lightweight so that the encoder generates informative latents and handles most of the processing--3. **Text Embedding and Decoder**: This will be very standard, we will have a UL2 setup with a tokenizer and embedding layer, and then a standard transformer decoder. The text dataset we will be using is TBD (maybe FineWebEdu). We will have a minor adjustment to UL2, instead of predicting just the noised parts we will reconstruct the whole sequence, to match what the audio decoder is doing better.--4. **Adversarial Alignment**: The discriminator will be something like a 2 layer MLP mixer taking in the matrix valued latent and outputting a number. We should either do a MLP mixer or a pool and then MLP to properly handle the matrixness of the latent. We might have to do a hyperparameter sweep directed towards getting the best update parameters for the discriminator.--5. **Encoder**: The encoder will be a memory mixer like what we have implemented already.--6. **Audio Projector**: The audio projector strips non-linguistic information (prosody, speaker identity, noise) from the audio latent so that the projected output is distributionally aligned with the text latent. A full $d \times d$ linear map cannot reduce rank, so a bottleneck is preferred. Options:- - (a) Full linear: $W \in \mathbb{R}^{d \times d}$ — baseline- - (b) LoRA-style bottleneck: $W = AB$ where $A \in \mathbb{R}^{d \times a}$, $B \in \mathbb{R}^{a \times d}$, $a \ll d$ — forces rank-$a$ projection, acting as an information bottleneck. Sweep $a \in \{d/16, d/8, d/4, d/2\}$.- - (c) 2-layer MLP with bottleneck: $\text{GELU}(z_a W_1) W_2$ — adds nonlinearity if the split is not linearly separable-diff --git a/experiements/architecture.txt b/experiements/architecture.txtdeleted file mode 100644index 0879586..0000000--- a/experiements/architecture.txt+++ /dev/null@@ -1,390 +0,0 @@-================================================================================-Assignee: Noah Cylich & Justin Lee-Deadline: 7th March-================================================================================--Thesis: Three complementary compression axes; Matryoshka FFN subnetworks,-group-aligned block sparsity, and INT4 quantization-aware training. can be-jointly applied during training to produce a single checkpoint that is-deployable at multiple accuracy-efficiency operating points. Training once at-d_model=D with matryoshka loss at factors {1x, 2x, 4x, 8x, ...} yields a family-of models from one training run, each further compressible via sparsification.--================================================================================-SECTION 1 — EXPERIMENTAL AXES-================================================================================--Independent variables:- A. matryoshka shrink factor f ∈ {1, 2, 4, 8, 16}- B. Sparsity ratio s ∈ {0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}- C. Group size g ∈ {16, 32, 64, 128}- D. FFN activation ∈ {dReLU, SwiGLU, GeGLU}- E. Model scale ∈ {TOY-128, MAIN-1024}- F. QAT ∈ {off, int4-group32}--Dependent variables (measured at EACH matryoshka factor):- - Validation perplexity (TinyStories)- - Quantized validation perplexity (INT4)- - WikiText-2 perplexity- - LAMBADA accuracy- - HellaSwag accuracy- - ARC-Easy accuracy- - Estimated parameter count at deployment dim- - Throughput (tokens/sec on inference)- - Bigram repetition rate (generation diversity)- - Block sparsity (% of fully-zero groups, group_size=g)- - Effective FLOPs (est. params × density)--================================================================================-SECTION 2 — MODEL CONFIGURATIONS-================================================================================--Architecture: MemoryMixer encoder + Transformer decoder, shared embedding,-GQA (grouped query attention), ZC-RMSNorm, RoPE, SentencePiece BPE (8192).--TOY-128 (rapid iteration):- mat_factors=[2, 4], d_model=128, num_heads=4, num_kv_heads=2,- num_enc_layers=2, num_dec_layers=2, d_ff=512,- max_enc_len=128, max_dec_len=128, vocab=8192, dtype=bfloat16- needle train --toy --seed {42,137,256}--MAIN-1024 (paper results):- mat_factors=[1, 2, 4, 8, 16], d_model=1024, num_heads=4,- num_kv_heads=2, num_enc_layers=4, num_dec_layers=2, d_ff=4096,- max_enc_len=256, max_dec_len=256, vocab=8192, dtype=bfloat16- needle train --seed {42,137,256}--Both trained on TinyStories (1M samples, encoder-decoder split at 30%).-Optimizer: Muon (kernels) + AdamW (embeddings/norms), WSD schedule.-All experiments report metrics at each matryoshka factor slice.--================================================================================-SECTION 3 — EXPERIMENT GROUPS-================================================================================--All experiments use 3 seeds (42, 137, 256) for statistical significance.-Report mean +/- std on all metrics. Every run automatically reports val PPL-and INT4-quantized val PPL at each matryoshka factor slice.----------------------------------------------------------------------------------GROUP 1: Dense Baselines + Matryoshka Factor Scaling (Table 1, Figure 1)---------------------------------------------------------------------------------Purpose: Establish quality at each deployment dimension from a single training-run. This is the matryoshka baseline — no sparsification.-- 1a. MAIN-1024, 3 epochs, no pruning, matryoshka factors [1024,512,256,128,64]- needle train --sparsity-ratio 0.0 --seed {42,137,256}-- 1b. TOY-128, 3 epochs, no pruning, matryoshka factors [128,64]- needle train --toy --sparsity-ratio 0.0 --seed {42,137,256}--Deliverables:- - Table: val PPL and quant PPL at each matryoshka factor (1024, 512, 256, 128, 64)- - Line plot: deployment dim vs val PPL — the matryoshka scaling curve (Figure 1a)- - Line plot: deployment dim vs estimated param count (Figure 1b)- - Benchmark scores at each dim- - These are the denominators for all relative comparisons----------------------------------------------------------------------------------GROUP 2: matryoshka + Group-Wise Sparsification (Table 2, Figure 2)- *** CORE CONTRIBUTION ***---------------------------------------------------------------------------------Purpose: Characterise how matryoshka factor scaling interacts with block sparsity.-At smaller deployment dims, does sparsification hurt more or less?--Schedule: Epoch 1 dense, epochs 2-3 with sparsity mask locked.-- For s in {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8}:- 2a. MAIN-1024, 3 epochs, --sparsity-ratio s- needle train --sparsity-ratio {s} --seed {42,137,256}-- 2b. Select best 2 sparsity ratios from 2a, run with 4 epochs (recovery)- needle train --epochs 4 --sparsity-ratio {s} --seed {42,137,256}--Deliverables:- - 2D heatmap: (sparsity, matryoshka factor) → val PPL (Figure 2a)- - Pareto curve per matryoshka factor: sparsity % vs val PPL (Figure 2b)- - Table: all metrics at each (sparsity, dim) pair- - Compression spectrum: for best s, show the range of models:- 1x (full) → 16x (smallest) with estimated params and PPL- - Analysis: does sparsification degrade smaller matryoshka slices more?- (Hypothesis: smaller slices are more sensitive because fewer dimensions- carry the representation — each zeroed weight matters more)- - Analysis: multiplicative compression — matryoshka factor reduction ×- sparsity = total compression ratio----------------------------------------------------------------------------------GROUP 3: Recovery Epoch Scaling (Table 3, Figure 3)---------------------------------------------------------------------------------Purpose: How many recovery epochs are needed after sparsification?--Pick the best sparsity ratio from Group 2a. Vary total epochs:-- For E in {2, 3, 4, 5, 6}:- 3a. MAIN-1024, E epochs, --sparsity-ratio {s}- (sparsification at epoch 1; recovery = E - 1 epochs)--Deliverables:- - Line plot per matryoshka factor: recovery epochs vs val PPL (Figure 3)- - Analysis: do smaller matryoshka factors need more recovery?----------------------------------------------------------------------------------GROUP 4: QAT Interaction Study (Table 4, Figure 4)---------------------------------------------------------------------------------Purpose: Show that group-wise sparsification is compatible with INT4 QAT-because both operate on the same group_size=32 granularity. Measure at-each matryoshka factor — INT4 quantization error at smaller dims may compound.-- 4a. Dense + QAT (baseline, = Group 1a)- 4b. Sparsified (s=0.3) + QAT- 4c. Sparsified (s=0.6) + QAT-- For comparison, also run 4b-4c WITHOUT QAT by modifying _train_step- to not call _quantize_params.--Deliverables:- - Table: (QAT on/off) × sparsity × matryoshka factor → val PPL- - Analysis: does QAT degrade more at smaller matryoshka factors?- (Hypothesis: yes — fewer dimensions means each quantization error- has proportionally larger impact on output quality)- - Effective compression: matryoshka factor reduction × sparsity × INT4 → total- bits per effective parameter----------------------------------------------------------------------------------GROUP 5: Group Size Ablation (Table 5, Figure 5)---------------------------------------------------------------------------------Purpose: Determine optimal block granularity for sparsification.--Fixed: MAIN-1024, s=0.3, 3 epochs, QAT on.-- For g in {16, 32, 64, 128}:- 5a. needle train --sparsity-ratio 0.3 --group-size {g} \- --seed {42,137,256}-- High sparsity stress test:- For g in {16, 32, 64, 128}:- 5b. needle train --sparsity-ratio 0.6 --group-size {g} \- --seed {42,137,256}--Deliverables:- - Line plot per matryoshka factor: group size vs val PPL (Figure 5a)- - Table: all metrics at each (group size, matryoshka factor)- - Effective bits-per-param: bpp = (1 - sparsity) * (4 + 32/g)- - Analysis: does optimal group size change with matryoshka deployment dim?----------------------------------------------------------------------------------GROUP 6: Activation Function x Sparsification Interaction (Table 6, Figure 6)- *** KEY FINDING ***---------------------------------------------------------------------------------Purpose: Test whether activation function choice determines sparsification-robustness across matryoshka factor slices.--Activation definitions:- - dReLU: ReLU(gate) * ReLU(up) — naturally ~75% sparse activations- - SwiGLU: SiLU(gate) * up — smooth, dense activations- - GeGLU: GELU(gate) * up — smooth, nearly dense activations-- 6a. Dense baselines (no sparsification) — isolate activation quality- For act in {drelu, swiglu, geglu}:- needle train --activation {act} --sparsity-ratio 0.0 \- --seed {42,137,256}-- 6b. Sparsification — test robustness per activation- For act in {drelu, swiglu, geglu}, s in {0.3, 0.6}:- needle train --activation {act} --sparsity-ratio {s} \- --seed {42,137,256}--Deliverables:- - Grouped bar chart per matryoshka factor: activation × compression → val PPL (Fig 6a)- - Relative degradation: (sparse_ppl / dense_ppl) per activation per dim- (Figure 6b) — KEY CHART: sparsification robustness not just raw quality- - Analysis: does dReLU's natural activation sparsity help or hurt at- smaller matryoshka factors where redundancy is already low?----------------------------------------------------------------------------------GROUP 7: Matryoshka Factor-Specific Analysis (Table 7, Figure 7)- *** matryoshka-SPECIFIC CONTRIBUTION ***---------------------------------------------------------------------------------Purpose: Deep-dive into what the model learns at each matryoshka factor slice.--Using best checkpoint from Group 2a:-- 7a. Representation similarity across dims- - CKA (centered kernel alignment) between 1x and each larger factor- - Measure: do smaller dims learn a subspace of the full representation?-- 7b. Per-dim generation quality- - Generate from each matryoshka factor slice at inference time- - Compare generation diversity, coherence, repetition rate-- 7c. matryoshka loss weighting ablation- - Uniform weighting (current): L = (1/K) * sum(L_k)- - Scaled weighting: L = sum(d_k/D * L_k) (larger dims weighted more)- - Inverse weighting: L = sum(D/d_k * L_k) (smaller dims weighted more)--Deliverables:- - CKA heatmap: dim × dim similarity (Figure 7a)- - Table: generation quality metrics at each dim- - Ablation table: matryoshka loss weighting strategy × dim → val PPL----------------------------------------------------------------------------------GROUP 8: Inference Efficiency Validation (Table 8, Figure 8)---------------------------------------------------------------------------------Purpose: Translate theoretical compression into real speedup across the-full spectrum of deployment options.--For each Pareto-optimal checkpoint from Groups 1-2, at each matryoshka factor:- needle test --checkpoint {path} --throughput-runs 50--Measure at each deployment dim:- - Tokens/sec (decode throughput)- - Latency per generation (ms)- - Peak memory (via JAX profiler)--Deliverables:- - Scatter plot: effective params vs actual throughput (Figure 8a)- - Table: all Pareto-optimal (s, factor) with throughput + quality- - The "menu" figure: one chart showing ALL deployment options from a- single checkpoint, from largest (1x, no pruning) to smallest- (16x, s=0.80) — THIS IS THE PAPER'S HERO FIGURE--================================================================================-SECTION 4 — EVALUATION PROTOCOL-================================================================================--For every checkpoint, evaluate at EACH matryoshka factor slice:-- 1. Validation perplexity (TinyStories val split)- - Full-precision val PPL- - INT4-quantized val PPL (at each matryoshka factor)- needle test --checkpoint {path} --max-eval-samples 5000-- 2. Benchmark suite (at full dim and smallest viable dim)- needle evaluate --checkpoint {path} \- --benchmarks wikitext2 lambada hellaswag arc_easy --max-samples 500-- 3. Generation quality at each matryoshka factor- needle test --checkpoint {path}- Report: bigram repetition rate, avg generation length, sample outputs-- 4. Throughput at each matryoshka factor- needle test --checkpoint {path} --throughput-runs 50--================================================================================-SECTION 5 — EXPECTED TABLES AND FIGURES-================================================================================--Tables:- T1. Dense baselines + matryoshka scaling curve (val PPL at each dim)- T2. matryoshka + sparsification (sparsity × dim → val PPL)- T3. Recovery epoch scaling (epochs × matryoshka factors)- T4. QAT interaction (sparsity × {on,off} × matryoshka factors)- T5. Group size ablation (4 sizes × matryoshka factors)- T6. Activation × sparsification interaction (3 act × compression × matryoshka factors)- T7. matryoshka factor analysis (CKA, generation quality, loss weighting)- T8. Inference efficiency (Pareto-optimal checkpoints × matryoshka factors)--Figures:- F1. matryoshka scaling curve: deployment dim vs PPL + param count- F2. matryoshka + sparsity: (a) heatmap, (b) Pareto per dim — KEY FIGURE- F3. Recovery epoch scaling per matryoshka factor- F4. QAT interaction: sparsity × QAT × matryoshka factor- F5. Group size: (a) g vs PPL per dim, (b) effective bpp- F6. Activation × sparsification: (a) grouped bars per dim, (b) degradation- F7. matryoshka representation analysis: (a) CKA heatmap, (b) gen quality vs dim- F8. Hero figure: full deployment menu from single checkpoint--================================================================================-SECTION 6 — COMPUTE BUDGET-================================================================================--Per-run cost (MAIN-1024, 3 epochs, 1M samples, 4x TPU v5 lite):- ~15 minutes training + ~5 minutes matryoshka evaluation (5 dims × val + quant)--Estimated total runs:- Group 1: 2 configs × 3 seeds = 6 runs- Group 2: 8 ratios × 3 seeds + 2 ratios × 3 seeds (rec)= 30 runs- Group 3: 5 epoch counts × 3 seeds = 15 runs- Group 4: 3 methods × 2 (QAT on/off) × 3 seeds = 18 runs- Group 5: 4 sizes × 2 sparsity × 3 seeds = 24 runs- Group 6: 3 act × (3 + 6) × 3 seeds = 27 runs- Group 7: 3 weighting strategies × 3 seeds = 9 runs- Group 8: evaluation only (reuses checkpoints) = 0 runs- ────────────────────────────────────────────────────────────────────- Total: 129 runs--At ~20 min/run: ~43 TPU-hours (manageable on v5litepod-4)--================================================================================-SECTION 7 — ABLATION CHECKLIST-================================================================================--Before submission, verify these are addressed:-- [ ] Statistical significance: 3 seeds, report mean +/- std- [ ] matryoshka scaling: smooth degradation curve from full dim to smallest- [ ] matryoshka + sparsity composability: both axes combine gracefully- [ ] Matched comparisons: compare methods at SAME effective param count- [ ] Recovery saturation: diminishing returns on fine-tuning epochs- [ ] QAT compatibility: sparsity + quantization + matryoshka are not antagonistic- [ ] Generation quality at each matryoshka factor, not just perplexity- [ ] Group size sensitivity: optimal g, interaction with matryoshka factor- [ ] Activation × sparsification × matryoshka factor: three-way interaction- [ ] matryoshka representation analysis: CKA, subspace structure- [ ] Honest limitations: sparse matmuls don't accelerate on current JAX;- matryoshka factor reduction requires custom kernel for real speedup- [ ] Comparison to baselines: training K separate small models vs one matryoshka--================================================================================-SECTION 8 — EXECUTION ORDER-================================================================================--Phase 1 (Day 1): Groups 1 + 2 (36 runs, ~12 hrs)- Baselines and sparsification sweeps, all with matryoshka evaluation.- Deliverable: matryoshka scaling curves and sparsity Pareto frontiers.--Phase 2 (Day 2): Groups 3 + 4 (33 runs, ~11 hrs)- Recovery and QAT ablations.- Deliverable: complete ablation tables.--Phase 3 (Day 3): Groups 5 + 6 (51 runs, ~17 hrs)- Group size and activation interaction studies.- Deliverable: granularity and activation robustness analysis.--Phase 4 (Day 4): Groups 7 + 8 + writing (9 runs + eval)- matryoshka analysis, inference benchmarks, paper draft.- Deliverable: hero figure, all tables populated.--================================================================================-SECTION 9 — PAPER NARRATIVE-================================================================================--Framing: "Train once, deploy anywhere."--A single training run with Matryoshka loss produces a checkpoint that can be-deployed at any matryoshka factor. Each sub-model can be further compressed with-group-wise sparsification. The result is a deployment MENU:-- ┌─────────────────────────────────────────────────────────────────────┐- │ Deployment option Est. params Val PPL Quant PPL │- │ ───────────────────────────────────────────────────────────────── │- │ 1x, full ~44M X.XX X.XX │- │ 2x, full ~11M X.XX X.XX │- │ 2x, s=0.3 ~7M X.XX X.XX │- │ 2x, s=0.6 ~4M X.XX X.XX │- │ 4x, full ~3M X.XX X.XX │- │ 4x, s=0.3 ~2M X.XX X.XX │- │ 8x, full ~0.8M X.XX X.XX │- │ 8x, s=0.5 ~0.4M X.XX X.XX │- │ 16x, full ~0.2M X.XX X.XX │- │ 16x, s=0.8 ~0.04M X.XX X.XX │- └─────────────────────────────────────────────────────────────────────┘--Key claims:- 1. Matryoshka FFN shrinking is nearly free — minimal degradation at 2x reduction- 2. Sparsification composes multiplicatively with matryoshka- 3. A single checkpoint spans 100x+ parameter range- 4. INT4 QAT is compatible with both compression axes- 5. MemoryMixer encoder is more compressible than standard transformer encoder--================================================================================diff --git a/experiements/datasets.txt b/experiements/datasets.txtdeleted file mode 100644index e1ac4dc..0000000--- a/experiements/datasets.txt+++ /dev/null@@ -1,27 +0,0 @@-================================================================================--Task: Design & gather multimodal dataset, text tokenization, speech processing, and tool call strategy.-Deliverable: A pipleine of preprocessed dataset, from which the codebase can stream datasets from.-Assignee: Parkirat Sandhu & Jakub Mroz-Deadline: 7th March--================================================================================--Text dataset: SYNTH (https://huggingface.co/datasets/PleIAs/SYNTH)-Transcribe datasets: Emilia-Large, SPGI Speech-Text tool call datasets: xlam-function-calling-60k, xlam-irrelevance-7.5k, Salesforce/APIGen-MT-5k, APIGen-Irrelevant-5k, glaive-function-calling, Team-ACE/ToolACE-Voice tool call datasets: Use a good TTS model to generate voice commands from the tool call dataset prompts--================================================================================--Note:-1. LEAP makes sense as we wish the dataset to preprocess and pretokenize the inputs before injesting into the model.-2. We are training a cutom sentencepiece tokenizer with a small vocab size, given how constrained the decode is.-3. Requires solid cloud infra experience, latency in injestion will slow down training.-4. Can text to tools translate to voice to tools.-4. Record all obersvations, they will go into the paper!-5. Design the batching (mix of speech and text and tools) experimantally.-6. We should have a <transcribe> and <tools> tag for the two tasks we train on.--================================================================================-diff --git a/experiements/training.md b/experiements/training.mddeleted file mode 100644index 019ef3f..0000000--- a/experiements/training.md+++ /dev/null@@ -1,166 +0,0 @@-----Task: Design the training strategy & loss functions for training a transcribe+tools model.-Deliverable: A well-executed ablation of all strategies against SOTA.-Assignee: Karen Mosoyan & Satyajit Kumar-Deadline: 7th March-----Notes:-1. We use EMA of weights & this is a deterministic task.-2. What is the best contrastive learning? SiGLip or CLIP?-3. How do we pretrain the encoder? Reconstruction + Contrastive (speech, text, structured)-3. Can we improve the sparsification schedule?-4. How can we incorporate iimplicit VAD?-5. To allow simultaneous generation, use 2 small heads for: transcribe, json.-6. Accumulate gradients 2x for speech and audio?-----## What do we want to do?-We want to develop a training recipe for a model that will be able to take either audio or text into the model, and output either a transcription of the audio or an action (tool call output) result based on the audio or the text.--## What do we want the model to look like?-We want:-1. A shared encoder that can process audio *or* text-2. Lightweight decoder or decoders that will very quickly decode the encoder output into either output type-3. A fast encoder that scales well to longer inputs--## What decisions are we pretty sure about?-1. The encoder we will likely be using is the MemoryMixer. It has several nice properties we would like to use, namely O(N) scaling in input length, GEMM heavy inference, and constant sized output.-2. The final model will have two separate decoder heads -- one for transcription and one for tool call.-3. The decoders will likely be simple shallow transformers.-4. We are pretty sure about the data we will be using being a lot of labelled audio data, some unlabelled audio data, and tool call data with text -> tool call and synthetically created audio -> tool call data.--## What do we need to figure out?-1. How to leverage the data we have to train the model as well as possible?-2. How to make sure the model handles audio and text inputs equivalently for tool call head?--## Some structural decisions we have made that we are less sure about-1. We will need a pretraining stage and a posttraining stage. The pretraining stage will train the encoder to model both audio and text modalities and align them independently of downstream tasks. The posttraining stage will train the model on the downstream tasks of transcription and tool call.-2. The pretraining stage will be as described in [pretraining.md](pretraining.md).-3. The posttraining stage will be as described in [posttraining.md](posttraining.md).-4. Pretraining will have aggressive data augmentation for audio so that our audio data is not too clean going on so that the model is able to model noise as well as speech.-5. Posttraining will have two separate stages -- the first stage being learning transcription, and the second stage being learning tool calling.--## Why this setup?-1. We want to have a pretraining stage so that the encoder starts off by learning to generally model text and audio jointly and the latents created by the encoder are aligned over the full distributions of audio and text. A nice thing about the pretraining is that we can utilize any unlabelled speech data we have in this stage.-2. The pretraining has adversarial alignment because that gives us the ability to align distributions without pairs (which are needed in contrastive learning), and gives us the ability to align the full distributions rather than pairs. The adversarial minimax training is also very constrained/regularized by the reconstruction loss pressure for audio and text so we don't expect typical instability and mode collapse that may come with GANs.-3. In the pretraining stage we need to align audio and text latent output distributions, however audio reconstruction needs a lot more information than just the speech transcription or speech text, so the audio latents will naturally be richer than the text latents. Therefore we want a audio->text latent projection that strips away this extra information.-4. In the posttraining stage we need to learn a fundamental skill first, which is transcription, and then tool calling will be very dependent on the transcription. Therefore we want to learn transcription well first before starting tool call training.-5. Since in posttraining we will be using paired (audio;text) data we can use contrastive loss to align actual pairs of latents which is nicer and potentially more stable than the adversarial alignment even though it doesn't have the same distribution wide guarantee-6. For the tool calling we want perfect retrieval of tool names and argument names, so we are inputting the tool list directly into the tool decoder head.-7. For the tool decoder head we want to use constrained generation, and this works great with a separate head-8. We find in experimentation that a single head cannot do both text modeling and transcription so we fully divide the transcription head and the tool calling head.-9. The pretraining will have several artifacts, including the audio pre-encoder (embedding), text embeddings, shared encoder, audio projector, and text reconstruction head. We will be transferring all of these to the posttraining and the text reconstruction head will serve as the initialization for both the transcription and the tool call head.-------## Architectural Options--These are the component-level design choices that need to be decided experimentally. Each has a small set of options to ablate.--1. **Audio Projector Architecture**: The projector maps audio latents into the text-aligned subspace. A full $d \times d$ matrix cannot reduce rank, so it may fail to strip non-linguistic information. Options:- - (a) Full linear: $W \in \mathbb{R}^{d \times d}$ — baseline, no rank reduction- - (b) LoRA-style bottleneck: $W = AB$ where $A \in \mathbb{R}^{d \times a}$, $B \in \mathbb{R}^{a \times d}$, with $a \ll d$ — forces the projected latent into a rank-$a$ subspace, acting as an information bottleneck that discards prosody/speaker/noise. Sweep $a \in \{d/16, d/8, d/4, d/2\}$.- - (c) 2-layer MLP with bottleneck: $\text{GELU}(z_a W_1) W_2$ — adds nonlinearity if the linguistic/non-linguistic split is not linearly separable--2. **Audio Embedding**: See [pretraining.md](../docs/pretraining.md) item 1. Options:- - (a) Raw waveform → learned convolutions (Moonshine-style)- - (b) MEL spectrogram → convolutions (Whisper-style)- - (c) MEL spectrogram → patches (ViT-style)--3. **Discriminator Architecture**: The discriminator takes in a matrix-valued latent $\in \mathbb{R}^{M \times d}$ and outputs a scalar. Options:- - (a) MLP-Mixer: treats slots as a sequence, mixes across both slot and hidden dimensions- - (b) Pool + MLP: global average pool over slots → 2-layer MLP- - (c) Learned query attention: single learned query attends to all slots → MLP on the attended output--4. **Tool Call Head Initialization via LoRA Induction Pre-training**: Before training the tool call head on real tool data, we pre-train it on synthetic structured retrieval tasks using a LoRA adapter on top of the frozen text reconstruction head weights. This preserves text generation capability while adding the retrieval/copying circuit. See [posttraining.md](../docs/posttraining.md) Stage 0 for details. Options:- - (a) LoRA adapter (rank $r$), merge before tool training — retrieval circuit becomes part of base weights- - (b) Full fine-tune on synthetic data — risk of overwriting text reconstruction capabilities- - (c) No induction pre-training — baseline-------## Training Schedule--### Pretraining Schedule-- **Data matching**: To prevent the discriminator from learning to distinguish modalities based on information density rather than content, we use density-matched batching. Audio clips of duration $t$ are batched with text passages of comparable semantic content ($\approx 2.5t$ words). This ensures padding ratios and energy distributions are matched across modalities within each batch, forcing the discriminator to align on content rather than structure.-- **Adversarial $\lambda_\text{adv}$ schedule**: Options include constant, linear warmup, or cyclical. The discriminator update frequency relative to the generator also needs to be set (e.g. $n_D$ discriminator steps per generator step).-- **Curriculum for audio augmentation**: Start with clean audio, gradually introduce noise augmentation (additive noise, reverb, speed perturbation) over the course of training.--### Posttraining Schedule-- **Stage 0 — Induction Head Pre-training**: Symbolic retrieval curriculum on the tool call head (see [posttraining.md](../docs/posttraining.md)). Curriculum:- - Phase 0a (~500 steps): 2 tools, 1 arg each, explicit values, fully random token embeddings- - Phase 0b (~500 steps): 5 tools, 2–3 args, explicit values, random embeddings- - Phase 0c (~1000 steps): 10 tools, 2–5 args, values embedded in natural context, noise $\sigma$ decaying from 1.0 → 0.3- - Phase 0d (~500 steps): Full complexity, light embedding noise $\sigma = 0.1$, transitioning to real embeddings-- **Stage 1 — Transcription training**: Audio-text paired data with SigLip contrastive alignment-- **Stage 2 — Tool call training**: Text→tool first, then audio→tool--### Module Transfer & LR/Freezing Ablations (Pretrain → Posttrain)-When transferring pretrained modules to posttraining, we need to decide per-module learning rate and freeze schedules:--| Module | Options |-|--------|---------|-| Audio embedding | Freeze for $N$ steps / reduced LR / full LR |-| Text embedding | Freeze for $N$ steps / reduced LR / full LR |-| Shared encoder | Freeze for $N$ steps / reduced LR / full LR |-| Audio projector | Freeze for $N$ steps / reduced LR / full LR |-| Transcription head (init from text recon) | Full LR (new task) |-| Tool call head (init from text recon + LoRA merge) | Full LR (new task) |--Ablation: Compare (a) freeze all transferred modules for first 10% of posttraining steps, (b) reduced LR (0.1× base) for transferred modules, (c) full LR everywhere. Measure catastrophic forgetting of alignment via discriminator probe and downstream task quality.-------## Macro Ablation Tree--The ablations below are ordered hierarchically. We run them top-down — negative results at a higher level prune the entire subtree below, saving compute.--### Level 0: Does pretraining help at all?-**Ablation 0.1**: Full pipeline (pretrain → posttrain) vs. posttrain only (randomly initialized encoder + heads)--If pretraining shows no benefit → skip all Level 1 and Level 2 ablations. Proceed directly to Level 3.--### Level 1: Does alignment in pretraining matter?-*Only run if Level 0 shows pretraining helps.*--**Ablation 1.1**: Pretrain with alignment (adversarial) vs. pretrain without alignment (reconstruction only, no discriminator)--If alignment shows no benefit → skip all Level 2 ablations.--### Level 2: Alignment & projector details-*Only run if Level 1 shows alignment helps.*--**Ablation 2.1**: Adversarial vs. contrastive alignment in pretraining (contrastive requires pseudo-pairs from an off-the-shelf ASR transcription)--**Ablation 2.2**: Audio projector architecture (full linear vs. LoRA bottleneck vs. MLP) — see Architectural Options item 1--**Ablation 2.3**: Discriminator architecture (MLP-Mixer vs. pool+MLP vs. learned query) — see Architectural Options item 3--**Ablation 2.4**: Density-matched batching vs. random batching — does the discriminator cheat on information density?--**Ablation 2.5**: $\lambda_\text{adv}$ schedule (constant vs. warmup vs. cyclical)--**Ablation 2.6**: No projector--### Level 3: Posttraining structure-*Always run (independent of pretraining decisions).*--**Ablation 3.1**: Induction head pre-training (Stage 0) on vs. off — does the symbolic retrieval curriculum improve tool call accuracy?--**Ablation 3.2**: SigLip contrastive loss in posttraining on vs. off — does contrastive alignment during task training help or fight the heads?--**Ablation 3.3**: Sequential (transcription → tool) vs. joint (transcription + text-tool simultaneously) — does learning transcription first help tool calling?--**Ablation 3.4**: Audio tool call training timing — simultaneous with text tool call vs. separate final stage--### Level 4: Component-level fine-tuning-*Run after settling Levels 0–3.*--**Ablation 4.1**: Audio embedding choice (waveform conv vs. MEL conv vs. MEL patches) — see Architectural Options item 2--**Ablation 4.2**: Module transfer LR/freezing strategy — see Training Schedule table--**Ablation 4.3**: Induction head curriculum details — random embeddings vs. fixed, decaying noise vs. constant noise, LoRA rank--**Ablation 4.4**: Gradient accumulation ratio for speech vs. text batches--**Ablation 4.5**: EMA decay $\beta$ schedule (constant vs. warmup)diff --git a/scripts/build_dataset.py b/scripts/build_dataset.pydeleted file mode 100644index 6e52425..0000000--- a/scripts/build_dataset.py+++ /dev/null@@ -1,164 +0,0 @@-#!/usr/bin/env python3-"""Build text-only tool-call dataset and save locally.--Usage:- python scripts/build_dataset.py- python scripts/build_dataset.py --max-samples 1000- python scripts/build_dataset.py --output data/tool_calls_unified-"""--import argparse-import json-import logging-import os-import sys--_shm = "/dev/shm"-if os.path.isdir(_shm):- try:- _st = os.statvfs(_shm)- if _st.f_bavail * _st.f_frsize >= 200 * 1024**3:- _hf_cache = os.path.join(_shm, "needle_hf_cache")- os.makedirs(_hf_cache, exist_ok=True)- os.environ.setdefault("HF_DATASETS_CACHE", os.path.join(_hf_cache, "datasets"))- except OSError:- pass--_token_path = os.path.expanduser("~/.cache/huggingface/token")-if os.path.isfile(_token_path) and "HF_TOKEN" not in os.environ:- with open(_token_path) as _f:- _tok = _f.read().strip()- if _tok:- os.environ["HF_TOKEN"] = _tok--import numpy as np--sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))-sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))-from tools_data import load_and_combine--logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")-logger = logging.getLogger(__name__)---def _default_output():- """Use /dev/shm for output when RAM is plentiful."""- shm = "/dev/shm"- if os.path.isdir(shm):- try:- st = os.statvfs(shm)- if st.f_bavail * st.f_frsize >= 200 * 1024**3:- return os.path.join(shm, "needle_data", "tool_calls_unified")- except OSError:- pass- return "data/tool_calls_unified"---def main():- parser = argparse.ArgumentParser(description="Build text-only tool-call dataset")- parser.add_argument("--max-samples", type=int, default=None, help="Max samples to include")- parser.add_argument("--output", type=str, default=_default_output(), help="Output directory")- args = parser.parse_args()-- logger.info("Loading and combining text datasets...")- combined = load_and_combine()-- if args.max_samples:- combined = combined.select(range(min(args.max_samples, len(combined))))-- logger.info(f"Dataset: {len(combined)} rows")-- os.makedirs(os.path.dirname(args.output) or ".", exist_ok=True)- combined.save_to_disk(args.output)- logger.info(f"Saved {len(combined)} rows to {args.output}/")-- # Persist to GCS- from src.gcs import upload_raw_data- upload_raw_data(args.output)-- print_summary(combined)---def print_summary(ds, n_samples=5):- """Print dataset statistics and sample examples."""- n = len(ds)-- # Parse tools/answers counts- n_tools, n_calls = [], []- n_with_calls, n_no_calls = 0, 0- query_lens, tools_lens, answers_lens = [], [], []- for ex in ds:- try:- tools = json.loads(ex["tools"])- except (json.JSONDecodeError, TypeError):- tools = []- try:- answers = json.loads(ex["answers"])- except (json.JSONDecodeError, TypeError):- answers = []- n_tools.append(len(tools))- n_calls.append(len(answers))- if answers:- n_with_calls += 1- else:- n_no_calls += 1- query_lens.append(len(ex["query"]))- tools_lens.append(len(ex["tools"]))- answers_lens.append(len(ex["answers"]))-- n_tools = np.array(n_tools)- n_calls = np.array(n_calls)- query_lens = np.array(query_lens)- tools_lens = np.array(tools_lens)- answers_lens = np.array(answers_lens)-- def _stats(arr):- return f"min={arr.min()}, max={arr.max()}, mean={arr.mean():.1f}, median={np.median(arr):.0f}"-- print(f"\n{'=' * 60}")- print(f"DATASET SUMMARY")- print(f"{'=' * 60}")- print(f" Total examples: {n:,}")- print(f" With tool calls: {n_with_calls:,} ({n_with_calls/n*100:.1f}%)")- print(f" No-call (empty): {n_no_calls:,} ({n_no_calls/n*100:.1f}%)")- print()- print(f" Available tools/sample: {_stats(n_tools)}")- print(f" Tool calls/sample: {_stats(n_calls)}")- print()- print(f" Query length (chars): {_stats(query_lens)}")- print(f" Tools length (chars): {_stats(tools_lens)}")- print(f" Answers length (chars): {_stats(answers_lens)}")-- # Samples with tool calls- with_calls_idx = [i for i in range(n) if n_calls[i] > 0]- no_calls_idx = [i for i in range(n) if n_calls[i] == 0]-- rng = np.random.RandomState(42)-- if with_calls_idx:- sample_idx = rng.choice(with_calls_idx, size=min(n_samples, len(with_calls_idx)), replace=False)- print(f"\n{'─' * 60}")- print(f"SAMPLES WITH TOOL CALLS ({len(sample_idx)})")- print(f"{'─' * 60}")- for i, idx in enumerate(sample_idx):- ex = ds[int(idx)]- print(f"\n [{i+1}] Query: {ex['query']}")- print(f" Tools: {ex['tools']}")- print(f" Answers: {ex['answers']}")-- if no_calls_idx:- sample_idx = rng.choice(no_calls_idx, size=min(n_samples, len(no_calls_idx)), replace=False)- print(f"\n{'─' * 60}")- print(f"SAMPLES WITHOUT TOOL CALLS ({len(sample_idx)})")- print(f"{'─' * 60}")- for i, idx in enumerate(sample_idx):- ex = ds[int(idx)]- print(f"\n [{i+1}] Query: {ex['query']}")- print(f" Tools: {ex['tools']}")- print(f" Answers: {ex['answers']}")-- print(f"\n{'=' * 60}\n")---if __name__ == "__main__":- main()