The best ChatGPT that $100 can buy.
Python
Commits on master
use nn.init and initialize smear gate's weight as well
9822cc7 Sofie Van Landeghem committed 7 months ago
fix: initialize smear and backout lambdas in init_weights
94b73ad Marcin Bogdanski committed 7 months ago
create a group for dev dependencies, there is no need to install all this other stuff just for speedrun and it's exposing people to dependency chain attacks. we need to delete more dependencies. dependencies bad bad bad
a445144 Andrej Karpathy committed 8 months ago
fix: use meta device in disable_fp8 to avoid VRAM spike (#616)
47e983e RoomWithOutRoof committed 8 months ago
use COMPUTE_DTYPE-aware cast in Muon polar express step
c0dbf1f Andrej Karpathy committed 8 months ago
bunch of ideas tried from openai/parameter-golf, all negative results for nanochat
4e1694c Andrej Karpathy committed 8 months ago
fix scaling laws scripts after the bigram embeddings were removed
5019acc Andrej Karpathy committed 8 months ago
submit new time to GPT-2 leaderboard entry: 99 minutes
1b1cc3c Andrej Karpathy committed 8 months ago
Autoresearch round 2: smear, backout, and hyperparameter tuning
a825e63 Andrej Karpathy committed 8 months ago