A straightforward method for training your LLM, from downloading data to generating text.
site_name: Train LLM From Scratch
site_description: From pretraining to RLHF and GRPO, every algorithm hand-written in pure PyTorch, on a GPU or a laptop CPU.
site_url: https://FareedKhan-dev.github.io/train-llm-from-scratch/
repo_url: https://github.com/FareedKhan-dev/train-llm-from-scratch
repo_name: FareedKhan-dev/train-llm-from-scratch
docs_dir: docs
# Files present under docs/ but intentionally not page nodes in the nav.
not_in_nav: |
/diagrams/README.md
# Keep the raw mermaid sources and the plotting script out of the built site.
exclude_docs: |
/diagrams/src/
/diagrams/*.py
theme:
name: material
icon:
repo: fontawesome/brands/github
palette:
- media: "(prefers-color-scheme: light)"
scheme: default
primary: teal
accent: teal
toggle: {icon: material/weather-night, name: Switch to dark mode}
- media: "(prefers-color-scheme: dark)"
scheme: slate
primary: teal
accent: teal
toggle: {icon: material/weather-sunny, name: Switch to light mode}
features:
- navigation.tabs
- navigation.sections
- navigation.top
- navigation.indexes
- toc.follow
- search.suggest
- search.highlight
- content.code.copy
- content.code.annotate
plugins:
- search
markdown_extensions:
- admonition
- attr_list
- md_in_html
- footnotes
- toc:
permalink: true
- pymdownx.highlight:
anchor_linenums: true
- pymdownx.inlinehilite
- pymdownx.arithmatex:
generic: true
- pymdownx.snippets:
base_path: ["."] # so howto/commands.md can include the root POST_TRAINING.md
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
format: !!python/name:pymdownx.superfences.fence_code_format
- pymdownx.tabbed:
alternate_style: true
extra_javascript:
- https://unpkg.com/mermaid@11/dist/mermaid.min.js # live hand-drawn mermaid in <details>
- javascripts/mathjax.js
- https://unpkg.com/mathjax@3/es5/tex-mml-chtml.js
nav:
- Home: README.md
- Laptop Track: student/README.md
- Foundations:
- Overview: foundations/README.md
- Tokenization & Data Shapes: foundations/tokenization.md
- BPE From Scratch: foundations/bpe.md
- Decoder-Only Transformer: foundations/transformer.md
- Attention, Masks & Heads: foundations/attention.md
- Objectives, Losses & Perplexity: foundations/objectives.md
- Optimization & Training Systems: foundations/optimization.md
- Generation & Sampling: foundations/generation.md
- Scaling, FLOPs & Memory: foundations/scaling.md
- Theory & Pipeline:
- Data handling: 01_data_pipeline.md
- Pretraining: 02_pretraining.md
- SFT: 03_sft.md
- Reward Model: 04_reward_model.md
- DPO / ORPO / KTO: 05_dpo.md
- PPO: 06_ppo.md
- GRPO / RLVR: 07_grpo.md
- Modern LLM:
- Overview: modern/README.md
- RMSNorm, SwiGLU & Init: modern/blocks.md
- Rotary Embeddings: modern/rope.md
- GQA, MLA & the KV Cache: modern/attention.md
- Mixture of Experts: modern/moe.md
- AdamW, Muon & Schedules: modern/optimizers.md
- DPO, IPO, SimPO, ORPO, KTO: modern/preference.md
- GRPO, Dr. GRPO, DAPO, GSPO: modern/rl_reasoning.md
- LoRA: modern/lora.md
- Fast Inference: modern/inference.md
- How-to:
- Configure (JSON): howto/configs.md
- Train (UI & CLI): howto/train.md
- Hardware (CPU, GPU, Mac, TPU): howto/hardware.md
- Type Safety: howto/type_safety.md
- The control-panel UI: howto/ui.md
- Evaluate: 08_evaluation.md
- Inference / Chat: 09_inference.md
- Reference:
- Command cheatsheet: howto/commands.md