Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
updates
HenryNdubuakuc53580f8de6d
diff --git a/experiements/training.txt b/experiements/training.txtindex 7243197..c01ed75 100644--- a/experiements/training.txt+++ b/experiements/training.txt@@ -7,8 +7,9 @@ Deadline: 7th MarchNotes:1. We use EMA of weights & this is a deterministic task.2. What is the best contrastive learning? SiGLip or CLIP?-3. How do we pretrain the encoder?+3. How do we pretrain the encoder? Reconstruction + Contrastive (speech, text, structured)3. Can we improve the sparsification schedule?4. How can we incorporate iimplicit VAD?5. To allow simultaneous generation, use 3 small heads for: transcribe, json.+6. Accumulate gradients 2x for speech and audio?================================================================================\ No newline at end of filediff --git a/src/cli.py b/src/cli.pyindex 04da4ce..5c32413 100644--- a/src/cli.py+++ b/src/cli.py@@ -26,9 +26,9 @@ HELP = """│ --sparsity-ratio FLOAT Block prune ratio (default: 0.33) ││ --layer-prune-ratio FL Layer prune ratio (default: 0.0) ││ --group-size INT Quant/prune group size (default: 32) │- │ --prune-interval INT Steps between mask updates (def: 100) │- │ --prune-start-frac FL Start pruning at this frac (def: 0.33) │- │ --prune-end-frac FL Lock mask at this frac (def: 0.67) │+ │ --prune-interval INT Steps between mask updates (def: 100) │+ │ --prune-start-frac FL Start pruning at this frac (def: 0.33)│+ │ --prune-end-frac FL Lock mask at this frac (def: 0.67) ││ --activation STR drelu|swiglu|geglu (default: drelu) ││ --warmup-ratio FLOAT LR warmup ratio (default: 0.05) ││ --eval-every INT Val eval interval (default: 1000) │