The best ChatGPT that $100 can buy.
Python
Commits on master
unify the two optimizer implementations into one and prevent bugs like the one that just happened where i updated one and didnt update the other
a9d0a86 Andrej Karpathy committed 4 months ago
nuke huggingface and its giant dependency footprint from orbit. datasets still todo
4186540 Andrej Karpathy committed 4 months ago
delete spurious 'cute' bloat that doesn't matter to a serious research stack
950a1dc Andrej Karpathy committed 4 months ago
remove a ton of UI bloat, we need to make nanochat small and sexy and minimal and forkable. code is ~free now, if you want UI get your favorite LLM to add it. nukes ~1000 LOC from orbit nice
da32e1d Andrej Karpathy committed 4 months ago
add a comment clarifying that RoPE here rotates by -theta (transpose of the textbook convention), which is functionally equivalent. Prompted by #489, thank you @qdrk
3616a02 Andrej Karpathy committed 4 months ago
fix token_bytes calculation to use raw token bytes instead of round-tripping through str, which corrupts tokens that are not valid standalone UTF-8 (e.g. bytes >= 0x80 became 3-byte U+FFFD). 193/32768 tokens were affected, so bpb of new runs will read ~0.05% higher than old runs. Thank you @ellenjxu #693
2ce972a Andrej Karpathy committed 4 months ago
docs: note that MPS handles bf16 fine on recent macOS (opt in via NANOCHAT_DTYPE=bfloat16, ~25% memory savings). Thank you @eaglstun #793 and @ifsheldon #685
5e43f1e Andrej Karpathy committed 4 months ago
allocate the inference KV cache in COMPUTE_DTYPE instead of hardcoding cuda->bf16, fixing bf16 inference (e.g. NANOCHAT_DTYPE=bfloat16 on MPS) and deleting an old hack. Thank you @mparrett #795
ca366eb Andrej Karpathy committed 4 months ago
fix mixed-dtype ops in the optimizer so that bf16 params (wte, value_embeds) work on MPS, which unlike CUDA refuses to promote dtypes. AdamW math now runs explicitly in fp32; verified bit-identical on CUDA, where inductor was already upcasting in registers. Thank you @mparrett for the report and investigation in #741
e02b339 Andrej Karpathy committed 4 months ago
adding MuonEq and Muon+, both seem to be very slight improvements at both d12 and d24. contributed by @jn2clark
4e014d3 Andrej Karpathy committed 4 months ago
delete the whole report thing i think it was a bad idea it just bloats the code. doing this subtracts 540 LOC
f10bd75 Andrej Karpathy committed 4 months ago
Merge pull request #686 from marcinbogdanski/fix/init-smear-backout-lambdas
b9b6ce1 Andrej committed 7 months ago