The best ChatGPT that $100 can buy.
Python
Commits on master
update the default GPTConfig kwargs otherwise they are confusing
e7ed208 Andrej Karpathy committed 10 months ago
update the CPU/MPS script to give reasonable results. The model can at least answer that Paris is the capital of France and knows that the sky is blue, for about 40 minutes of training on my macbook. Also fixed a bug that existed due to KVCache bfloat16 dtype assumption
f9a7e0f karpathy committed 10 months ago
add detection of device to report more correct mfu for bf16
2955650 Andrej Karpathy committed 10 months ago
Add high value engine tests for core invariants (33 LoC) (#396)
bbc4413 Barış Özmen committed 10 months ago
Reduce token waste in BOS bestfit by cropping shortest doc (#445)
e1dafc5 Yamahammer committed 10 months ago
naturally i failed to include the actual code in the previous commit facepalm
b62a5bc Andrej Karpathy committed 10 months ago
implement flash attention 3 fallback to pytorch sdpa by touching as few lines of code as possible in main files and keeping all implementation to a single file. add tests. add helpful warning messages for the user.
8203efa Andrej Karpathy committed 10 months ago
fuse adamw into a single torch compiled kernel similar to muon. it's about 1.7X faster, but overall it's so tiny that it's not making a major dent
22a71aa Andrej Karpathy committed 10 months ago
changes and optimizations to muon, making it more efficient and simpler/cleaner a bit
6bb9240 Andrej Karpathy committed 10 months ago
fix buggy midtrain and update all kwargs to be idiomatic. that is, argparse uses dashes variables use underscores. the underscores are just a remnant of the previous Configurator object. This is the right way
7312ec9 Andrej Karpathy committed 10 months ago
fix base_loss to report correct loss by switching the dataloader to the new default
3b50b77 Andrej Karpathy committed 10 months ago