Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

better wsd schedule

HenryNdubuaku7fecc431e024
diff --git a/src/train.py b/src/train.py
index 2d015cd..14c6dcf 100644
--- a/src/train.py
+++ b/src/train.py
@@ -121,7 +121,7 @@ def _wsd_schedule(peak_value, total_steps, warmup_steps, decay_ratio=0.15):
[
optax.linear_schedule(0.0, peak_value, warmup_steps),
optax.constant_schedule(peak_value),
- optax.cosine_decay_schedule(peak_value, decay_steps, alpha=0.0),
+ optax.cosine_decay_schedule(peak_value, decay_steps, alpha=0.05),
],
boundaries=[warmup_steps, warmup_steps + stable_steps],
)