The best ChatGPT that $100 can buy.
Python
Commits on master
fix minor bug in fp8 application to skip tiny matmuls
bac5a35 Andrej Karpathy committed 9 months ago
tune the data mixture a bit, load optimizer by default when SFT. These were confirmed to be best settings from sweeps of sft
1415fb7 Andrej Karpathy committed 9 months ago
a number of upgrades to SFT script to bring it up to date w.r.t. pretraining and tuning some of its kwargs based on sweeps
77f8fb8 Andrej Karpathy committed 9 months ago
tune the data mixture a bit, load optimizer by default when SFT. These were confirmed to be best settings from sweeps of sft
8180e1d Andrej Karpathy committed 9 months ago
Replace cryptic assertion with descriptive ValueError for batch size alignment
0f3b6a4 suraj-self committed 9 months ago
a number of upgrades to SFT script to bring it up to date w.r.t. pretraining and tuning some of its kwargs based on sweeps
788dade Andrej Karpathy committed 9 months ago
Save FP8 tensors in autograd ctx instead of full-precision inputs
d9678ff Alan committed 9 months ago
delete torchao dependency, create our own exact API-matched version of Float8Linear, document it very well. for some poorly understood reason, the performance is not only ~identical but actually runs 3% faster. despite of it being significantly simpler and much less code. i don't fully understand why/how atm
e569b59 Andrej Karpathy committed 9 months ago
tune miniseries just a bit, fairly cosmetic, keep to even depths where the math works out nicely in model sizing
ff46300 Andrej Karpathy committed 9 months ago
better comments/flow on all the hyperparameter transfer stuff, and change the WD scaling from my empirical 1/d^2 to a bit more principled version based on Tepoch. All of that theory is based on AdamW and could be suboptimal for Muon
aeff095 Andrej Karpathy committed 9 months ago
briefly mention batch ramp experimentation too, too weak to merge in my few attempts
e527521 Andrej Karpathy committed 9 months ago