Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.
better wsd schedule
HenryNdubuaku7fecc431e024
diff --git a/src/train.py b/src/train.pyindex 2d015cd..14c6dcf 100644--- a/src/train.py+++ b/src/train.py@@ -121,7 +121,7 @@ def _wsd_schedule(peak_value, total_steps, warmup_steps, decay_ratio=0.15):[optax.linear_schedule(0.0, peak_value, warmup_steps),optax.constant_schedule(peak_value),- optax.cosine_decay_schedule(peak_value, decay_steps, alpha=0.0),+ optax.cosine_decay_schedule(peak_value, decay_steps, alpha=0.05),],boundaries=[warmup_steps, warmup_steps + stable_steps],)