The best ChatGPT that $100 can buy.
Python
Commits on master
add negative result about not allowing attention across BOS tokens. A lot more code complexity for basically no gain in performance
f92efce Andrej Karpathy committed 10 months ago
Big DataLoader refactor: BOS-aligned dataloaders with epoch tracking for pre/mid-training
43c29dd Andrej Karpathy committed 10 months ago
adjust the comment on the regex pattern per recent experimnet see dev/LOG.md
2398541 Andrej Karpathy committed 10 months ago
validated that \p{N}{1,2} is the correct number of digits to group up to in the regex pattern of the GPT-4 tokenizer (2 down from 3), leading to the best val_bpb for 32K vocabs
64b48d0 Andrej Karpathy committed 10 months ago
document my struggle with fp8 integration yesterday, it's not working like i thought it would and i suffered. one day i will return to continue the fight.
238353c Andrej Karpathy committed 10 months ago
allow base_loss to report the loss of any arbitrary huggingface model similar to base_eval. had to change dataloader to be a lot better and just take tokenizer, not load the nanochat one. much better this way anyway
21608ec Andrej Karpathy committed 10 months ago
make miniseries more generic and easier to run and less hard coded
aa95fb2 Andrej Karpathy committed 10 months ago
add alternating window size patterns for the GPT layers, following GPT-3. Experimented a bit and found the pattern SSSL to work well - 3 short, 1 long alternating. This is now the new default and the plots look quite a bit better on flops vs. bpb
fbc1484 Andrej Karpathy committed 10 months ago
integrate Flash Attention 3. +9% tok_per_sec for d12 with ctx even as low as 2048 out of the box nice. also, ready to tune windows huge
2ff7d51 Andrej Karpathy committed 10 months ago
recover the ability to load old checkpoints by patching the lambdas if they don't exist in checkpoints
201d705 Andrej Karpathy committed 10 months ago
Add learnable lambdas that gate the residual connection and a skip connection to the input embeddings, solid bump to val_bpb
aa530cd Andrej Karpathy committed 10 months ago
fix adamw slight bug. this chunk was copy pasted originally from modded-nanogpt, which still seems to have the bug
4ddc803 Andrej Karpathy committed 10 months ago
remove rust compilation as rustbpe is now installed from separate package (#416)
a1ccb3d Sofie Van Landeghem committed 10 months ago
delete grad_clip. appears to not be necessary at all. not only was it buggy because the clipping happened per gpu before grad synchronization, but it costs ~2% MFU, and it also doesn't even help. I tried deleting it a while ago and back then it did help. So I'm guessing that some hyperparameter tuning obviated the reason for it since then
061f83c Andrej Karpathy committed 10 months ago
also add scaling_laws.sh script if it's a useful reference
3af4dcf Andrej Karpathy committed 10 months ago
nudge hyperparameters of the base script with the results of the sweeps and miniseries. vocab size down to 32K. D:N ratio from 20 to 8. add miniseries script
ccf4b7f Andrej Karpathy committed 10 months ago
Fix undefined variable in chat_rl after recent refactor
1b5de29 Adria Blancafort committed 10 months ago
tune hyperparameters based on overnight sweeps. warmdown_ratio is the biggest free win, increasing 0.2 -> 0.4, and embedding lr can be larger bumping 0.2 -> 0.3
ae0bf52 Andrej Karpathy committed 10 months ago
also add matplotlib dep so that we can have jupyter notebooks
eec0c79 Andrej Karpathy committed 10 months ago
add notebook on deriving the CORE estimates for the GPT-3 miniseries.
54e59c3 Andrej Karpathy committed 10 months ago
many small fixes to base_train: reporting ETA, allowing some additional kwarg flexibility, making sure we don't crash when e.g. depth = 11 - we now calculate the closest num_heads that works
9d4c9b7 Andrej Karpathy committed 10 months ago
alright add transformers as a dep of the repo because it should be easy to evaluate the CORE score of HF models. Not super happy about it but i tried it and the uv.lock doesn't get bloated as much as i expected
962b6bf Andrej Karpathy committed 10 months ago
delete the configurator in favor of argparse and clean up a lot of kwarg details to make them more consistent across all scripts
eb7bbc1 Andrej Karpathy committed 10 months ago