Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

updates

HenryNdubuakuc53580f8de6d
diff --git a/experiements/training.txt b/experiements/training.txt
index 7243197..c01ed75 100644
--- a/experiements/training.txt
+++ b/experiements/training.txt
@@ -7,8 +7,9 @@ Deadline: 7th March
Notes:
1. We use EMA of weights & this is a deterministic task.
2. What is the best contrastive learning? SiGLip or CLIP?
-3. How do we pretrain the encoder?
+3. How do we pretrain the encoder? Reconstruction + Contrastive (speech, text, structured)
3. Can we improve the sparsification schedule?
4. How can we incorporate iimplicit VAD?
5. To allow simultaneous generation, use 3 small heads for: transcribe, json.
+6. Accumulate gradients 2x for speech and audio?
================================================================================
\ No newline at end of file
diff --git a/src/cli.py b/src/cli.py
index 04da4ce..5c32413 100644
--- a/src/cli.py
+++ b/src/cli.py
@@ -26,9 +26,9 @@ HELP = """
│ --sparsity-ratio FLOAT Block prune ratio (default: 0.33) │
│ --layer-prune-ratio FL Layer prune ratio (default: 0.0) │
│ --group-size INT Quant/prune group size (default: 32) │
- │ --prune-interval INT Steps between mask updates (def: 100) │
- │ --prune-start-frac FL Start pruning at this frac (def: 0.33) │
- │ --prune-end-frac FL Lock mask at this frac (def: 0.67) │
+ │ --prune-interval INT Steps between mask updates (def: 100) │
+ │ --prune-start-frac FL Start pruning at this frac (def: 0.33)│
+ │ --prune-end-frac FL Lock mask at this frac (def: 0.67) │
│ --activation STR drelu|swiglu|geglu (default: drelu) │
│ --warmup-ratio FLOAT LR warmup ratio (default: 0.05) │
│ --eval-every INT Val eval interval (default: 1000) │