Skip to content

Build a modern LLM from scratch. Every line commented. Explained like we are five.

Commits on master

beam search
raiyanyahya committed 4 months ago
9203628
Add funding information for GitHub
Raiyan Yahya committed 4 months ago
d97213c
adding MOE and speculative decoding
raiyanyahya committed 5 months ago
ae560cc
adding flash attention and ho to read loss
raiyanyahya committed 5 months ago
00c355b
group query attention
raiyanyahya committed 5 months ago
b7a6ae1
adding fint tuning
raiyanyahya committed 6 months ago
cd57c88
fix readme
raiyanyahya committed 6 months ago
04132fd
af415bc
MPS after CUDA .. thanks Peter
raiyanyahya committed 6 months ago
725f664
update readme
raiyanyahya committed 6 months ago
cb66684
simplify redundant amp checks, thanks peter
raiyanyahya committed 6 months ago
6e96c22
a complete story
raiyanyahya committed 6 months ago
705ad92
a tokens journey
raiyanyahya committed 6 months ago
396131c
adding more explanation and examples chapters
raiyanyahya committed 6 months ago
561b808
add bpe and embeddings
raiyanyahya committed 6 months ago
f6c7e65
a little attention is all we need :)
raiyanyahya committed 6 months ago
12d2109
RoPe explanation and small example
raiyanyahya committed 6 months ago
d13ec26
fix line count to reflect the new actual count
raiyanyahya committed 6 months ago
ed4858f
fixes and updates
raiyanyahya committed 6 months ago
b54621f
fix the sqrt scaling explanation
raiyanyahya committed 6 months ago
d176d03
use cpu torch by defeault
raiyanyahya committed 6 months ago
3ae628a
put everything in a main py file
raiyanyahya committed 6 months ago
94cc202
train and infer
raiyanyahya committed 6 months ago
00f00cc
evrything included works
raiyanyahya committed 6 months ago
63e8cf7