The best ChatGPT that $100 can buy.
Python
Commits on master
keep varunneal kernel on H100, use community kernel for other supported cuda architectures
683b88c svlandeg committed 8 months ago
new leaderboard entry coming from improvements of autoresearch round 1, time to gpt-2 from 2.02 hours to 1.80 hours
f068604 Andrej Karpathy committed 8 months ago
All of these improvements were developed by Claude running autonomously over ~2 days using autoresearch. I didn't touch anything - incredible. All tuning was done on d12 but generalized easily to larger models (e.g. d24 in particular). This means we will also get a new "Time to GPT-2" Leaderboard entry, which I will push separately.
6ed7d1d Andrej Karpathy committed 8 months ago
Extend informative assertion message to chat_sft.py for consistency
f2899a1 suraj-self committed 8 months ago
delete autocast, an unnecessary thorn in my side, manage dtypes directly
1076f97 Andrej Karpathy committed 8 months ago
Ensure that inputs and targets are contiguous (#569)
752abc8 Sofie Van Landeghem committed 8 months ago
Document new Leaderboard entry congrats @ddudek for pointing out ClimbMix, time to GPT-2 is now 2.01 hours, down from 2.76 previously
4b40774 Andrej Karpathy committed 8 months ago
big, breaking change but large upside: swap previous FineWeb-EDU dataset to NVIDIA ClimbMix dataset. Requires people to download the data shards. The upside is that training GPT-2 capablity model now only takes ~2 hours, down from 2.76 hours, so this is a huge win data-wise
324e69c Andrej Karpathy committed 8 months ago
document the legacy fineweb100b dataset and the new climbmix400b dataset
b07604e Andrej Karpathy committed 8 months ago
remove inherited parameters from commandline and replace by model-tag
00a562c svlandeg committed 8 months ago
Merge remote-tracking branch 'upstream/master' into fix-batch-size-assertion
d489a1f suraj-self committed 9 months ago