Skip to content

A straightforward method for training your LLM, from downloading data to generating text.

docs(pipeline): drop em dashes from the stage pages

FareedKhan-dev73e4fce75de3
diff --git a/docs/01_data_pipeline.md b/docs/01_data_pipeline.md
index 403acc2..ca75f66 100644
--- a/docs/01_data_pipeline.md
+++ b/docs/01_data_pipeline.md
@@ -2,7 +2,7 @@
# Data Handling & Preprocessing
Every stage needs data in a different shape, and getting these shapes right is honestly half the
-battle — a misaligned loss mask or a mis-parsed gold answer will silently wreck training. So before
+battle: a misaligned loss mask or a mis-parsed gold answer will silently wreck training. So before
any model code, here is exactly how I download and preprocess each dataset, and the format each
trainer expects.
@@ -105,7 +105,7 @@ PYTHONPATH=. python scripts/prepare_sft_data.py --context_length 1024 --out_dir
```
I verified on the real file that the mask covers exactly `<answer>4</answer>` and excludes the user
-question — that alignment is what makes SFT work.
+question. That alignment is what makes SFT work.
## 3 · Preference data (→ `{prompt, chosen, rejected}` JSONL)
@@ -122,7 +122,7 @@ def _split_hh(text):
Output is `preferences.jsonl` (train) + `preferences_test.jsonl` (held-out, for measuring reward-model
accuracy). [`preference_dataset.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/data_loader/preference_dataset.py) tokenizes each side through
-the same chat template and right-pads a batch — which is safe because the model's attention is
+the same chat template and right-pads a batch, which is safe because the model's attention is
**causal**, so the last real token never attends to padding after it (no attention mask needed).
```bash
@@ -133,7 +133,7 @@ PYTHONPATH=. python scripts/prepare_preference_data.py --source both --max_per_s
[`prepare_rl_prompts.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/prepare_rl_prompts.py) turns GSM8K into prompts with a **verifiable
numeric gold answer** (parsed from the dataset's `#### N`), plus a programmatic **arithmetic curriculum**
-that even a weak policy can partly solve — so RL has non-zero reward signal to bootstrap from:
+that even a weak policy can partly solve, so RL has non-zero reward signal to bootstrap from:
```python
gold = gsm8k_gold_answer(ex["answer"]) # the number after '####'
@@ -144,7 +144,7 @@ rows.append({"prompt": ex["question"].strip(), "gold": gold})
PYTHONPATH=. python scripts/prepare_rl_prompts.py --out_dir /ephemeral/data
```
-I cross-checked the emitted gold answers 50/50 against the live GSM8K dataset — they match exactly,
+I cross-checked the emitted gold answers 50/50 against the live GSM8K dataset. They match exactly,
which matters because the verifier reward ([08_evaluation.md](08_evaluation.md)) is only as trustworthy
as the gold it compares against.
@@ -159,4 +159,4 @@ as the gold it compares against.
| `arithmetic_prompts.jsonl` | `{prompt, gold}` | GRPO curriculum warm-up |
<br>
-➡️ Next: [Stage 1 — Pretraining](02_pretraining.md).
+➡️ Next: [Stage 1: Pretraining](02_pretraining.md).
diff --git a/docs/02_pretraining.md b/docs/02_pretraining.md
index b8cd08a..7faa5b5 100644
--- a/docs/02_pretraining.md
+++ b/docs/02_pretraining.md
@@ -1,12 +1,12 @@
<!-- omit in toc -->
-# Stage 1 — Pretraining the base model
+# Stage 1: Pretraining the base model
Everything downstream is only as good as the base model, so the first thing I do is pretrain a
**~400M-parameter** version of this repo's own `Transformer` from scratch on the Pile. The original
[`train_transformer.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/train_transformer.py) is a clean single-GPU loop; for a mid-size
model on 2×H100 I wrote [`pretrain_base.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/pretrain_base.py), which adds the few things
-that actually matter at this scale — DistributedDataParallel, bf16 autocast, gradient accumulation, a
-cosine LR schedule with warmup, and periodic checkpointing — without touching the model itself.
+that actually matter at this scale (DistributedDataParallel, bf16 autocast, gradient accumulation, a
+cosine LR schedule with warmup, and periodic checkpointing) without touching the model itself.
If the architecture or training terms are unfamiliar, read the foundations chapters first:
[Decoder-Only Transformer](foundations/transformer.md),
@@ -66,11 +66,11 @@ optimizer.step()
A few choices worth calling out:
- **bf16 autocast** ([`amp_autocast`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/utils.py)) needs no `GradScaler` (unlike
fp16), which keeps the loop clean. Master weights stay fp32.
-- **AdamW with a weight-decay split** ([`configure_optimizer`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/optim.py)) — decay
+- **AdamW with a weight-decay split** ([`configure_optimizer`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/optim.py)): decay
the 2-D weight matrices, never the biases / norms / embeddings (the standard GPT recipe).
-- **Cosine LR with warmup** ([`cosine_lr`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/optim.py)) — linear ramp for
+- **Cosine LR with warmup** ([`cosine_lr`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/optim.py)): linear ramp for
`warmup_steps`, then cosine decay to `min_lr`.
-- **DDP** ([`distributed.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/distributed.py)) — each rank seeds its data shuffle
+- **DDP** ([`distributed.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/distributed.py)): each rank seeds its data shuffle
differently so the two GPUs see different windows; only rank 0 logs and checkpoints.
## Run it
@@ -90,13 +90,13 @@ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True PYTHONPATH=. \
## What the numbers mean
-- **train_loss** — running cross-entropy; starts near `ln(vocab) ≈ 10.8` and should fall steadily
+- **train_loss**: running cross-entropy; starts near `ln(vocab) ≈ 10.8` and should fall steadily
(mine went `11.06 → 8.6 → 6.0 → …`). This is the single best health signal.
-- **tok/s** — throughput (~32k/s combined on 2×H100 here).
-- **eval train/dev** — averaged loss on held-out windows, printed every `eval_steps`; watch the dev
+- **tok/s**: throughput (~32k/s combined on 2×H100 here).
+- **eval train/dev**: averaged loss on held-out windows, printed every `eval_steps`; watch the dev
loss to spot overfitting.
Checkpoints are written to `/ephemeral/ckpts/base_pretrained.pt` every `save_every` steps and carry the
config, so every later stage can rebuild the exact model with [`load_backbone_from_ckpt`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/utils.py).
-➡️ Next: [Stage 2 — SFT](03_sft.md).
+➡️ Next: [Stage 2: SFT](03_sft.md).
diff --git a/docs/03_sft.md b/docs/03_sft.md
index fb15a7a..bbb686d 100644
--- a/docs/03_sft.md
+++ b/docs/03_sft.md
@@ -1,5 +1,5 @@
<!-- omit in toc -->
-# Stage 2 — Supervised Fine-Tuning (SFT)
+# Stage 2: Supervised Fine-Tuning (SFT)
The base model can continue text but it doesn't know it's supposed to *answer* you. SFT fixes that by
showing it thousands of `(instruction, response)` pairs and training it to produce the **response**.
@@ -56,8 +56,8 @@ logits keeps the cross-entropy numerically clean under bf16.
## The trainer
[`train_sft.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/train_sft.py) loads the pretrained base with
-[`load_backbone_from_ckpt`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/utils.py), then runs a compact loop — autocast forward,
-masked loss, clip, step, cosine LR — with periodic dev evaluation:
+[`load_backbone_from_ckpt`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/utils.py), then runs a compact loop (autocast forward,
+masked loss, clip, step, cosine LR) with periodic dev evaluation:
```python
tokens, mask, epoch = next(train_it)
@@ -82,14 +82,14 @@ PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_sft.py # bot
## What the numbers mean
-- **train_loss / ppl** — masked cross-entropy (and its perplexity) over assistant tokens; should drop
+- **train_loss / ppl**: masked cross-entropy (and its perplexity) over assistant tokens; should drop
well below the base model's loss. To sanity-check the mechanics I ran an *overfit* test on 8 rows and
watched the loss collapse `11.0 → 4.7`, confirming the gradient path learns.
-- **dev_loss** — the same masked loss on a held-out split (`sft_dev_packed.h5`); the honest signal.
-- **GSM8K dev accuracy** — after SFT the model both follows instructions *and* emits the
+- **dev_loss**: the same masked loss on a held-out split (`sft_dev_packed.h5`); the honest signal.
+- **GSM8K dev accuracy**: after SFT the model both follows instructions *and* emits the
`<answer>…</answer>` format, so this should rise above the base model (see [08_evaluation.md](08_evaluation.md)).
The result is saved to `/ephemeral/ckpts/sft.pt` and becomes the starting point for the reward model,
DPO, PPO and GRPO.
-➡️ Next: [Stage 3 — Reward Model](04_reward_model.md) or jump to [DPO](05_dpo.md).
+➡️ Next: [Stage 3: Reward Model](04_reward_model.md) or jump to [DPO](05_dpo.md).
diff --git a/docs/04_reward_model.md b/docs/04_reward_model.md
index 892f2d5..db4969a 100644
--- a/docs/04_reward_model.md
+++ b/docs/04_reward_model.md
@@ -1,9 +1,9 @@
<!-- omit in toc -->
-# Stage 3 — Reward Model
+# Stage 3: Reward Model
To do classic RLHF (PPO) we need something that scores a response with a single number: higher = more
preferred. That's the reward model. I build it by putting a tiny scalar head on top of the SFT
-backbone and training it on human preference pairs with the **Bradley-Terry** loss — the same recipe
+backbone and training it on human preference pairs with the **Bradley-Terry** loss, the same recipe
as InstructGPT.
This page assumes you already know how the backbone produces hidden states. If not, start with
@@ -39,7 +39,7 @@ flowchart LR
[`RewardModel`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_model.py#L37) wraps a `Transformer`, drops the `lm_head`,
and reads the reward off the **last real token's** hidden state (the InstructGPT convention). Because
attention is causal, that last token has seen the whole sequence and never attends to the right-padding
-after it — so we need no attention mask:
+after it, so we need no attention mask:
```python
class RewardModel(nn.Module):
@@ -64,8 +64,8 @@ def bradley_terry_loss(chosen_rewards, rejected_rewards):
return -F.logsigmoid(chosen_rewards - rejected_rewards).mean()
```
-[`preference_accuracy`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_train.py#L23) — the fraction of pairs where
-`r_chosen > r_rejected` — is the metric I actually watch.
+[`preference_accuracy`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_train.py#L23), the fraction of pairs where
+`r_chosen > r_rejected`, is the metric I actually watch.
## The trainer
@@ -94,12 +94,12 @@ PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_reward.py
## What the numbers mean
-- **loss** — Bradley-Terry; starts at `-log σ(0) = 0.693` (chance) and drops as the gap widens.
-- **train_acc / test_acc** — preference accuracy. On clean fixtures it goes to `1.0`; on **real, noisy**
- HH-RLHF / UltraFeedback expect roughly **0.65–0.75** — that's normal, human preferences are noisy.
-- **margin** — mean `r_chosen − r_rejected`; a useful "is it still separating them" signal.
+- **loss**: Bradley-Terry; starts at `-log σ(0) = 0.693` (chance) and drops as the gap widens.
+- **train_acc / test_acc**: preference accuracy. On clean fixtures it goes to `1.0`; on **real, noisy**
+ HH-RLHF / UltraFeedback expect roughly **0.65 to 0.75**. That's normal: human preferences are noisy.
+- **margin**: mean `r_chosen − r_rejected`; a useful "is it still separating them" signal.
Saved to `/ephemeral/ckpts/reward.pt`; PPO loads it with
[`load_reward_model`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_model.py) when `--reward_source rm`.
-➡️ Next: [Stage 5 — PPO](06_ppo.md) (which consumes this), or the RM-free path: [DPO](05_dpo.md).
+➡️ Next: [Stage 5: PPO](06_ppo.md) (which consumes this), or the RM-free path: [DPO](05_dpo.md).
diff --git a/docs/05_dpo.md b/docs/05_dpo.md
index 52b9425..cd7165e 100644
--- a/docs/05_dpo.md
+++ b/docs/05_dpo.md
@@ -1,9 +1,9 @@
<!-- omit in toc -->
-# Stage 4 — DPO (and ORPO / KTO)
+# Stage 4: DPO (and ORPO / KTO)
Direct Preference Optimization is the shortcut around RLHF: instead of training a reward model and then
running an RL loop, DPO optimizes the policy *directly* on preference pairs, using a frozen copy of the
-SFT model as a reference anchor. No reward model, no rollouts, no value function — just one clean loss.
+SFT model as a reference anchor. No reward model, no rollouts, no value function: just one clean loss.
I also implemented two popular variants behind a `--loss_type` flag: **ORPO** (reference-free) and
**KTO** (works from a desirable/undesirable signal).
@@ -64,9 +64,9 @@ def dpo_loss(policy_chosen_logps, policy_rejected_logps, ref_chosen_logps, ref_r
The two **variants** ([`orpo_loss`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/dpo.py#L48),
[`kto_loss`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/dpo.py#L71)) live in the same file:
-- **ORPO** — reference-free: combines the SFT negative-log-likelihood on the chosen response with an
+- **ORPO**: reference-free: combines the SFT negative-log-likelihood on the chosen response with an
odds-ratio preference term, folding SFT + alignment into one stage (no frozen reference needed).
-- **KTO** — treats chosen as *desirable* and rejected as *undesirable* against a reference-KL baseline
+- **KTO**: treats chosen as *desirable* and rejected as *undesirable* against a reference-KL baseline
estimated from the batch; useful when you only have thumbs-up/down rather than pairs.
## The trainer
@@ -91,17 +91,17 @@ PYTHONPATH=. python scripts/train_dpo.py --loss_type orpo --orpo_lambda 1.0
PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_dpo.py
```
-> DPO uses a **small** learning rate (`5e-7` by default) — it's easy to over-push away from the
+> DPO uses a **small** learning rate (`5e-7` by default). It's easy to over-push away from the
> reference and degrade the model, so go gentle.
## What the numbers mean
-- **loss** — DPO/KTO start near `0.693`; ORPO starts higher (it includes the NLL term).
-- **acc** — implicit-reward accuracy: fraction of pairs where the policy's implicit reward prefers the
+- **loss**: DPO/KTO start near `0.693`; ORPO starts higher (it includes the NLL term).
+- **acc**: implicit-reward accuracy: fraction of pairs where the policy's implicit reward prefers the
chosen response. Should climb above 0.5.
-- **r_chosen / r_rejected** — the implicit rewards `β·(logπ − logref)`; the gap (margin) should widen.
-- **GSM8K dev accuracy** — the real downstream check.
+- **r_chosen / r_rejected**: the implicit rewards `β·(logπ − logref)`; the gap (margin) should widen.
+- **GSM8K dev accuracy**: the real downstream check.
Saved to `/ephemeral/ckpts/dpo.pt`.
-➡️ Next: the RL path — [PPO](06_ppo.md) and [GRPO](07_grpo.md).
+➡️ Next: the RL path, [PPO](06_ppo.md) and [GRPO](07_grpo.md).
diff --git a/docs/06_ppo.md b/docs/06_ppo.md
index 103c74c..e215a1a 100644
--- a/docs/06_ppo.md
+++ b/docs/06_ppo.md
@@ -1,8 +1,8 @@
<!-- omit in toc -->
-# Stage 5 — PPO (classic RLHF)
+# Stage 5: PPO (classic RLHF)
This is the original ChatGPT recipe: let the model generate, score the generations with a reward, and
-nudge the policy toward higher-reward behaviour using Proximal Policy Optimization — with a value
+nudge the policy toward higher-reward behaviour using Proximal Policy Optimization, with a value
network (critic) for variance reduction and a KL penalty to keep it from drifting too far from the SFT
model. I wrote the whole loop from scratch: rollout → reward → GAE advantages → clipped update.
@@ -39,7 +39,7 @@ flowchart LR
## The actor-critic
PPO needs a per-token value estimate `V(s_t)` next to the policy logits. I get both from one backbone
-with [`TransformerWithValueHead`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/value_head.py#L19) — it reuses
+with [`TransformerWithValueHead`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/value_head.py#L19). It reuses
`forward_hidden` + `lm_head` for the policy and adds a small scalar value head (initialized to ~0 so the
critic doesn't destabilize early training):
@@ -58,7 +58,7 @@ completion for each, and [`generate_with_logprobs`](https://github.com/FareedKha
sampling log-probs. Log-probs are always taken in **fp32** ([`compute_logprobs`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rollout.py#L233))
because PPO subtracts them and bf16 rounding there is harmful.
-## GAE — Generalized Advantage Estimation
+## GAE: Generalized Advantage Estimation
[`compute_gae`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/ppo.py#L24) works in the "action frame" (index `t` = producing
token `t+1`), bootstrapping only while the next action is still a response token:
@@ -101,16 +101,16 @@ PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_ppo.py
## What the numbers mean
-- **reward** — mean task reward per iteration; the headline curve, should trend up.
-- **KL_ref** — mean KL of the policy from the SFT reference; must stay **bounded**. If it blows up the
- model is degenerating — lower the LR or raise `--kl_coef`.
-- **clipfrac** — fraction of tokens hitting the PPO clip; a health/▒step-size signal.
-- **value_loss** — critic regression error.
-- **GSM8K test accuracy** — the real outcome, evaluated every `--eval_every`.
+- **reward**: mean task reward per iteration; the headline curve, should trend up.
+- **KL_ref**: mean KL of the policy from the SFT reference; must stay **bounded**. If it blows up the
+ model is degenerating: lower the LR or raise `--kl_coef`.
+- **clipfrac**: fraction of tokens hitting the PPO clip; a health and step-size signal.
+- **value_loss**: critic regression error.
+- **GSM8K test accuracy**: the real outcome, evaluated every `--eval_every`.
> PPO is the touchy one: small LR (`1e-6`), `clip 0.2`, grad-clip 1.0, and watch KL. I verified the loop
-> truly *optimizes* by giving it a learnable synthetic reward — reward climbed `0.10 → 1.00`.
+> truly *optimizes* by giving it a learnable synthetic reward: reward climbed `0.10 → 1.00`.
Saved to `/ephemeral/ckpts/ppo.pt`.
-➡️ Next: [Stage 6 — GRPO](07_grpo.md), which drops the critic entirely.
+➡️ Next: [Stage 6: GRPO](07_grpo.md), which drops the critic entirely.
diff --git a/docs/07_grpo.md b/docs/07_grpo.md
index 796aa5a..a0b9d46 100644
--- a/docs/07_grpo.md
+++ b/docs/07_grpo.md
@@ -1,10 +1,10 @@
<!-- omit in toc -->
-# Stage 6 — GRPO / RLVR (the reasoning frontier)
+# Stage 6: GRPO / RLVR (the reasoning frontier)
GRPO (Group Relative Policy Optimization) is the algorithm behind DeepSeek-R1, and it's beautifully
simple: **throw away PPO's value network**. For each prompt, sample a whole *group* of answers, score
them with a verifiable reward, and use the group's own mean/std as the baseline. The advantage is just
-"how much better than your groupmates was this answer?" — no critic to train, no value loss.
+"how much better than your groupmates was this answer?" No critic to train, no value loss.
For the group-relative advantage formula and how it relates to PPO-style policy ratios, see
[Objectives, Losses & Perplexity](foundations/objectives.md).
@@ -36,7 +36,7 @@ flowchart LR
## Group-relative advantage
-[`group_advantages`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/grpo.py#L17) is the whole idea — standardize rewards *within
+[`group_advantages`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/grpo.py#L17) is the whole idea: standardize rewards *within
each group*, so a good answer is one that beats its siblings on the same prompt:
```python
@@ -47,7 +47,7 @@ def group_advantages(rewards, group_size, eps=1e-4):
```
A nice property: if every answer in a group gets the same reward (all right or all wrong), the std-based
-advantage is ~0 and that group simply contributes no gradient — so I log the fraction of *informative*
+advantage is ~0 and that group simply contributes no gradient, so I log the fraction of *informative*
groups as a health metric.
## The loss: clipped surrogate + KL
@@ -87,11 +87,11 @@ PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_grpo.py
## What the numbers mean
-- **reward** — mean verifier reward across the group samples; the curve you want climbing.
-- **informative** — fraction of groups with non-zero reward spread (groups that actually teach
+- **reward**: mean verifier reward across the group samples; the curve you want climbing.
+- **informative**: fraction of groups with non-zero reward spread (groups that actually teach
something). If this collapses to 0, raise temperature / group size or stay longer on the curriculum.
-- **KL** — KL to the reference; keep it bounded.
-- **GSM8K test accuracy** — the headline reasoning metric, evaluated every `--eval_every`.
+- **KL**: KL to the reference; keep it bounded.
+- **GSM8K test accuracy**: the headline reasoning metric, evaluated every `--eval_every`.
> I verified the GRPO path genuinely optimizes: with a learnable reward the mean reward climbed
> **0.10 → 0.69 → 1.00** in ~15 iterations and saturated. PPO and GRPO share the same rollout/log-prob
diff --git a/docs/08_evaluation.md b/docs/08_evaluation.md
index a5c2941..8c2d1f2 100644
--- a/docs/08_evaluation.md
+++ b/docs/08_evaluation.md
@@ -3,8 +3,8 @@
A pipeline is only believable if you can measure it, so I evaluate every stage on the **same** held-out
GSM8K test set with greedy decoding. The headline deliverable is a single table: GSM8K accuracy as it
-moves Base → SFT → DPO → PPO → GRPO. The reward is *verifiable* — I parse the model's final number and
-compare it to the gold answer — so the score is objective, not a judgment call.
+moves Base → SFT → DPO → PPO → GRPO. The reward is *verifiable* (I parse the model's final number and
+compare it to the gold answer), so the score is objective, not a judgment call.
![Evaluation flow](diagrams/08_evaluation.png)
@@ -44,9 +44,9 @@ responses = batched_generate(model, prompts, max_new_tokens, device=device, gree
correct = sum(is_correct(resp, gsm8k_gold_answer(ans)) for (q, ans), resp in zip(qa_pairs, responses))
```
-The reward/checker lives in [`rewards/`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rewards/). `extract_answer` is tolerant —
+The reward/checker lives in [`rewards/`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rewards/). `extract_answer` is tolerant:
it prefers an `<answer>…</answer>` tag, then a GSM8K-style `#### N`, then falls back to the last number
-in the text — and [`reward_gsm8k`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rewards/verifiers.py#L35) is
+in the text. [`reward_gsm8k`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rewards/verifiers.py#L35) is
**correctness-dominant** with only a small, bounded format bonus, to discourage reward hacking:
```python
@@ -85,14 +85,14 @@ grpo ... 200
## In-training metrics
Each trainer also writes a metrics JSONL under `/ephemeral/logs/` (via
-[`MetricsLogger`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/logging_utils.py)) — train/dev loss for SFT, preference accuracy
+[`MetricsLogger`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/logging_utils.py)): train/dev loss for SFT, preference accuracy
for the reward model, implicit-reward accuracy for DPO, and reward/KL/clip-fraction + GSM8K accuracy for
PPO/GRPO. Pass `--use_wandb true` to also mirror to Weights & Biases; the JSONL is always written so you
can plot offline.
## What "good" looks like at this scale
-A ~400M from-scratch model won't top the GSM8K leaderboard — the point is the **relative** climb across
+A ~400M from-scratch model won't top the GSM8K leaderboard. The point is the **relative** climb across
stages and bounded KL during RL. Expect modest absolute numbers but a clear, real before/after gain at
each step.
diff --git a/docs/09_inference.md b/docs/09_inference.md
index 8e9df26..c566cdf 100644
--- a/docs/09_inference.md
+++ b/docs/09_inference.md
@@ -1,9 +1,9 @@
<!-- omit in toc -->
# Inference & Chat
-Training is only satisfying if you can actually *talk* to the result. The original
-[`generate_text.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/generate_text.py) does raw continuation for the base model, but it's
-hard-wired to the legacy config and has no chat template — so I added a small inference layer that loads
+Training is only satisfying if you can actually *talk* to the result.
+[`generate_text.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/generate_text.py) continues text with the models from `train_transformer.py`, but it
+has no chat template, so I added a small inference layer that loads
**any** stage checkpoint (base / SFT / DPO / PPO / GRPO) and talks to it correctly.
For the underlying decoding loop, context cropping, temperature, and stop-token behavior, read
@@ -49,9 +49,9 @@ state = {k.removeprefix("module.").removeprefix("transformer."): v for k, v in s
[`generate_reply`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/inference.py#L37) has two modes, reusing the same tested
generation core as training/eval ([`batched_generate`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/evaluation.py#L24)):
-- **chat** (default) — wraps your text in the chat template (optionally with a `system` message) and
+- **chat** (default): wraps your text in the chat template (optionally with a `system` message) and
returns the decoded assistant turn. Use this for SFT/DPO/PPO/GRPO checkpoints.
-- **raw** (`--raw`) — treats your text as a prefix and returns the base model's continuation (no
+- **raw** (`--raw`): treats your text as a prefix and returns the base model's continuation (no
template). Use this for `base_pretrained.pt`.
```python
@@ -63,7 +63,7 @@ out = batched_generate(model, [ids], max_new_tokens, device=device,
temperature=temperature, top_k=top_k, top_p=top_p, greedy=greedy)
```
-Decoding is defensive — [`decode`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/chat_template.py) drops the EOT terminator and
+Decoding is defensive: [`decode`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/chat_template.py) drops the EOT terminator and
any padding-vocab ids (the model's vocab is padded to 50304 but r50k_base only decodes 0–50255).
## The CLI
@@ -85,8 +85,8 @@ on `--device cuda` or `cpu` (both verified).
## Sampling knobs, briefly
-- **greedy** — reproducible, best for eval / math (`--greedy`).
-- **temperature** — higher = more random; ~`0.7–1.0` for open-ended chat.
-- **top_p / top_k** — nucleus / top-k truncation to cut the long tail of unlikely tokens.
+- **greedy**: reproducible, best for eval / math (`--greedy`).
+- **temperature**: higher = more random; ~`0.7–1.0` for open-ended chat.
+- **top_p / top_k**: nucleus / top-k truncation to cut the long tail of unlikely tokens.
That's the full loop: pretrain → align → reason → measure → chat. Back to the [overview](README.md).
diff --git a/docs/README.md b/docs/README.md
index 04db457..8f872b5 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -1,9 +1,9 @@
<!-- omit in toc -->
-# Post-Training & Alignment — Overview
+# Post-Training & Alignment: Overview
When I first trained this transformer from scratch, it could *continue* text but it couldn't
*follow instructions* or *reason*. That's what post-training fixes. This `docs/` folder walks
-through the whole journey I built on top of the base model — every stage written from scratch
+through the whole journey I built on top of the base model, every stage written from scratch
in plain PyTorch (no `trl`, no `peft`, no `transformers`), trained on real public datasets, and
runnable on a single GPU or scaled across multiple GPUs with DDP.
@@ -74,14 +74,14 @@ flowchart TD
| 4 | **DPO / ORPO / KTO** | to prefer better answers *without* an RL loop | [05_dpo.md](05_dpo.md) |
| 5 | **PPO** | to maximize a reward (RM or verifier) with the classic RLHF loop | [06_ppo.md](06_ppo.md) |
| 6 | **GRPO / RLVR** | to reason, using verifiable rewards (DeepSeek-R1 style) | [07_grpo.md](07_grpo.md) |
-| — | **Data pipeline** | how every dataset above is downloaded & preprocessed | [01_data_pipeline.md](01_data_pipeline.md) |
-| — | **Evaluation** | how I measure GSM8K accuracy across all stages | [08_evaluation.md](08_evaluation.md) |
-| — | **Inference / chat** | how to actually talk to any checkpoint | [09_inference.md](09_inference.md) |
+| | **Data pipeline** | how every dataset above is downloaded & preprocessed | [01_data_pipeline.md](01_data_pipeline.md) |
+| | **Evaluation** | how I measure GSM8K accuracy across all stages | [08_evaluation.md](08_evaluation.md) |
+| | **Inference / chat** | how to actually talk to any checkpoint | [09_inference.md](09_inference.md) |
## The one design rule: *wrap, don't rewrite*
Everything here sits on top of the original [`Transformer`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/models/transformer.py). I changed the
-educational model in exactly **one** place — I added a [`forward_hidden`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/models/transformer.py#L56)
+educational model in exactly **one** place: I added a [`forward_hidden`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/models/transformer.py#L56)
method that returns the final hidden states the `lm_head` consumes. Every post-training head (a value
head for PPO, a scalar reward head for the reward model) and every RL log-prob computation composes
*around* that one method, so the from-scratch model you already understand stays intact.
@@ -92,8 +92,8 @@ head for PPO, a scalar reward head for the reward model) and every RL log-prob c
· 🟧 RL / reward · 🟥 loss / objective · 🟪 evaluation · ⬜ checkpoint
> Each diagram is a hand-drawn, colour-coded Mermaid sketch, **pre-rendered to a PNG and embedded as
-> an image** (GitHub's live Mermaid doesn't reliably do `look: handDrawn`, and some viewers — e.g. the
-> VS Code preview — block SVGs, so an embedded PNG shows everywhere). The editable Mermaid source sits
+> an image** (GitHub's live Mermaid doesn't reliably do `look: handDrawn`, and some viewers, e.g. the
+> VS Code preview, block SVGs, so an embedded PNG shows everywhere). The editable Mermaid source sits
> in a collapsible *"Mermaid source"* block under each image. To regenerate the images after editing,
> see [diagrams/README.md](diagrams/README.md).