Skip to content

Automation foundation model for tiny devices: 2-bit, 8-29 MB, tool calls, ASR, structured extraction and embeddings on phones, wearables, smart homes, robots, cars and microcontrollers.

Commits on main

revert gated delta net
HenryNdubuaku committed 8 months ago
18ac111
strided convolution
HenryNdubuaku committed 8 months ago
3c885b4
Matryoshka export
HenryNdubuaku committed 8 months ago
fd873c3
Add experiments
HenryNdubuaku committed 8 months ago
126117f
updates
HenryNdubuaku committed 8 months ago
794d7fa
update instructions
HenryNdubuaku committed 8 months ago
f39c985
training
HenryNdubuaku committed 8 months ago
fd57d5f
reduce decoder size
HenryNdubuaku committed 8 months ago
b60241b
small vocab size
HenryNdubuaku committed 8 months ago
a9334ee
Training restart
HenryNdubuaku committed 8 months ago
c6fd91f
orthogonalisation
HenryNdubuaku committed 8 months ago
779293f
memoryMixer
HenryNdubuaku committed 8 months ago
e880863
remove gradient accumulation
HenryNdubuaku committed 8 months ago
961d001
embedding scaling
HenryNdubuaku committed 8 months ago
90d51ed
QAT
HenryNdubuaku committed 8 months ago
d3ca772
Sparsification
HenryNdubuaku committed 8 months ago
7ccad31
pruning techniques
HenryNdubuaku committed 8 months ago
3940814
Blocwise pruning
HenryNdubuaku committed 8 months ago
6d896e1
QAT
HenryNdubuaku committed 8 months ago
1fa0059
training report
HenryNdubuaku committed 8 months ago
2805085
Bug fixes
HenryNdubuaku committed 8 months ago
796a13e
Training changes
HenryNdubuaku committed 8 months ago
1ace71f
Change tokenizer
HenryNdubuaku committed 8 months ago
b616b4e
Zero-centred RMS Norm
HenryNdubuaku committed 8 months ago
812ae1f
GQA
HenryNdubuaku committed 8 months ago
2deb4f5
FIx default precision
HenryNdubuaku committed 8 months ago
a03639f
Improve baseline
HenryNdubuaku committed 8 months ago
c4b40ed
Optimise and stabilise train
HenryNdubuaku committed 8 months ago
7118112
TPU bug fixes
HenryNdubuaku committed 8 months ago
b297f00
updates
HenryNdubuaku committed 8 months ago
765bb7a