The best ChatGPT that $100 can buy.
Python
Commits on master
small bugfix make mid_train script work even with a tiny number of iterations
39cccc5 Andrej committed 11 months ago
clean up train/val loader in sft for consistency with mid/base
58f3e84 Andrej committed 11 months ago
Fix distributed Parquet dataloader resume for multi-epoch training
72a7cf2 Andrej committed 11 months ago
group common code to make things neater in gpt logit computation
bffdb2e Andrej Karpathy committed 11 months ago
apply float32 cast before logits softcapping so the tanh is in fp32. torch compile fuses this correctly with no extra memory costs.
cbf30c8 Andrej committed 11 months ago
fix bug where any rank has to be able to create checkpoint_dir if saving optim
90442de Andrej Karpathy committed 11 months ago
Fix distributed Parquet dataloader resume for multi-epoch training
01ea71b sunyujun03 committed 11 months ago
Fix kv_cache indexing to explicitly include head dimension
a770dce deepbuilder committed 11 months ago
Refactor find_last_step to use os.listdir with regex filtering
3e2a066 Pyry Takala committed 1 year ago
Fix find_last_step crash on checkpoint files with extra underscores
01f5f10 Pyry Takala committed 1 year ago
make mid_train script work even with a tiny number of iterations
dddb95c Eric Silberstein committed 1 year ago
renamed find_largest_model() argument checkpoint_dir to checkpoints_dir for clarity
a4a0959 Eric Silberstein committed 1 year ago
change test/train split approach because random.seed(1) and random.seed(-1) do the same thing
9777070 Eric Silberstein committed 1 year ago
Merge pull request #299 from samjabrahams/rotary_embedding_head_dim_comment_cleanup
4a87a0d Andrej committed 1 year ago