The best ChatGPT that $100 can buy.
Python
Commits on master
merge two files base_loss and base_eval into a single file, it's nicer this way, and unify the huggingface code associated with both
0307997 Andrej Karpathy committed 9 months ago
nuke midtraining from orbit, it's not as needed now that we have a BOS-aligned dataloader. Also change the README a lot. midtrianing is not yet fully properly erased across the board, but good enough for step 1
1ddaad1 Andrej Karpathy committed 9 months ago
fix dataloader for midtrain to never crop data. we can't just throw it away like we do in pretraining
348fbb3 Andrej Karpathy committed 9 months ago
Fix SDPA KV-cache decode to respect sliding window (#456)
3ba42e8 Andrej Karpathy committed 9 months ago
feat: allow top_k=0 in web api to disable filtering (#458)
ace6740 Aarushi Singh committed 9 months ago
i am feeling in a delete mood today. i need to delete a lot of code. there is too much code and surface area and complexity. ew
02baa15 Andrej Karpathy committed 9 months ago
i think this is the new torch 2.9+ API for declaring tf32 preference
d6c4f3b Andrej Karpathy committed 9 months ago
contiguous views and single HtoD transfer for inputs/targets much cleaner
6a341f2 Andrej Karpathy committed 9 months ago
tried muonh, appealing but didn't work out of the box
ebd4d9b Andrej Karpathy committed 9 months ago
Combine AdamW and Muon into single MuonAdamW optimizer, cleaner, ty @chrisjmccormick for idea/help
41bb2ea Andrej Karpathy committed 9 months ago
revert engram, not seeing an improvement at larger scale
74554be Andrej Karpathy committed 9 months ago
add engram-lite, add log, tune scaling laws analysis scripts
c8d93be Andrej Karpathy committed 9 months ago
first version of engram following modded nanogpt style
59e36cc Andrej Karpathy committed 10 months ago
320 experiments just to tune the adam beta1 of x0 a little bit up from 0.8 to 0.96
85b3e95 Andrej Karpathy committed 10 months ago
fix: pass device_type to compute_init in engine.__main__ (#451)
6a477ee xiayan0118 committed 10 months ago
something i've wanted to do for a while - move all .sh runs to their own directory so they don't pollute root dir
63bb583 Andrej Karpathy committed 10 months ago
resolve a crash for odd depths because FA3 needs head_dim % 8 == 0
cf5c9e5 Andrej Karpathy committed 10 months ago