A straightforward method for training your LLM, from downloading data to generating text.
Commits on main
docs(howto): drop em dashes from the how-to pages, POST_TRAINING.md and the diagrams README
af8561d FareedKhan-dev committed 1 month ago
fix(deps): add jaxtyping and regex to the requirements files
2605846 FareedKhan-dev committed 1 month ago
docs(howto): type safety with typed configs, jaxtyping shapes and mypy
5887c5b FareedKhan-dev committed 1 month ago
fix(config): show config paths with forward slashes in error messages on Windows
4a256da FareedKhan-dev committed 1 month ago
docs(howto): hardware guide for CPU, NVIDIA, Apple Silicon and TPU
3fa4cc4 FareedKhan-dev committed 1 month ago
test(tokenizer): the fast BPE trainer learns the same merges as the textbook algorithm
60b95be FareedKhan-dev committed 1 month ago
docs(modern): LoRA from scratch, with measured adapter sizes for both architectures
a3739c9 FareedKhan-dev committed 1 month ago
docs(modern): GRPO and the 2025 follow-ups (Dr. GRPO, DAPO, GSPO) with their flags
ce87b2a FareedKhan-dev committed 1 month ago
docs(modern): preference optimization with DPO, cDPO, IPO, SimPO, ORPO and KTO
d10e02f FareedKhan-dev committed 1 month ago
docs(modern): AdamW, Muon with Newton-Schulz, and cosine, WSD and linear schedules
53f317c FareedKhan-dev committed 1 month ago
docs(modern): Mixture of Experts routing, the balancing loss and shared experts
8b7a600 FareedKhan-dev committed 1 month ago
docs(modern): attention with GQA, MLA, QK-norm, sliding windows and the KV cache
797390c FareedKhan-dev committed 1 month ago
docs(modern): rotary position embeddings, with the math, the code and rope_theta
0ca046e FareedKhan-dev committed 1 month ago
docs(modern): RMSNorm, SwiGLU, tied embeddings and depth-scaled initialization
35d7b88 FareedKhan-dev committed 1 month ago
feat(inference): top-p and min-p in speculative decoding
2ee1a0e FareedKhan-dev committed 1 month ago
feat(models): top-p and min-p sampling in the classic model's generate
78d95fc FareedKhan-dev committed 1 month ago
feat(scripts): speculative decoding in generate_text.py with --draft_model
1c39df5 FareedKhan-dev committed 1 month ago
feat(inference): per-model context windows for speculative decoding
adad6bd FareedKhan-dev committed 1 month ago
docs(diagrams): make_plots.py, plots computed from the repo's own code
559d734 FareedKhan-dev committed 1 month ago
docs(diagrams): speculative decoding, BPE training and typed configs
4ac8b20 FareedKhan-dev committed 1 month ago
docs(diagrams): GRPO variants, the laptop student track and LoRA
c529fbc FareedKhan-dev committed 1 month ago
docs(diagrams): the modern decoder block, GQA, the KV cache and MoE
c71fc78 FareedKhan-dev committed 1 month ago
fix(models): stop saving the classic model's causal masks in checkpoints
b7bc603 FareedKhan-dev committed 1 month ago
feat(scripts): model_report.py for parameters, FLOPs, memory and training time
957c620 FareedKhan-dev committed 1 month ago