A straightforward method for training your LLM, from downloading data to generating text.
docs(pipeline): drop em dashes from the stage pages
FareedKhan-dev73e4fce75de3
diff --git a/docs/01_data_pipeline.md b/docs/01_data_pipeline.mdindex 403acc2..ca75f66 100644--- a/docs/01_data_pipeline.md+++ b/docs/01_data_pipeline.md@@ -2,7 +2,7 @@# Data Handling & PreprocessingEvery stage needs data in a different shape, and getting these shapes right is honestly half the-battle — a misaligned loss mask or a mis-parsed gold answer will silently wreck training. So before+battle: a misaligned loss mask or a mis-parsed gold answer will silently wreck training. So beforeany model code, here is exactly how I download and preprocess each dataset, and the format eachtrainer expects.@@ -105,7 +105,7 @@ PYTHONPATH=. python scripts/prepare_sft_data.py --context_length 1024 --out_dir```I verified on the real file that the mask covers exactly `<answer>4</answer>` and excludes the user-question — that alignment is what makes SFT work.+question. That alignment is what makes SFT work.## 3 · Preference data (→ `{prompt, chosen, rejected}` JSONL)@@ -122,7 +122,7 @@ def _split_hh(text):Output is `preferences.jsonl` (train) + `preferences_test.jsonl` (held-out, for measuring reward-modelaccuracy). [`preference_dataset.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/data_loader/preference_dataset.py) tokenizes each side through-the same chat template and right-pads a batch — which is safe because the model's attention is+the same chat template and right-pads a batch, which is safe because the model's attention is**causal**, so the last real token never attends to padding after it (no attention mask needed).```bash@@ -133,7 +133,7 @@ PYTHONPATH=. python scripts/prepare_preference_data.py --source both --max_per_s[`prepare_rl_prompts.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/prepare_rl_prompts.py) turns GSM8K into prompts with a **verifiablenumeric gold answer** (parsed from the dataset's `#### N`), plus a programmatic **arithmetic curriculum**-that even a weak policy can partly solve — so RL has non-zero reward signal to bootstrap from:+that even a weak policy can partly solve, so RL has non-zero reward signal to bootstrap from:```pythongold = gsm8k_gold_answer(ex["answer"]) # the number after '####'@@ -144,7 +144,7 @@ rows.append({"prompt": ex["question"].strip(), "gold": gold})PYTHONPATH=. python scripts/prepare_rl_prompts.py --out_dir /ephemeral/data```-I cross-checked the emitted gold answers 50/50 against the live GSM8K dataset — they match exactly,+I cross-checked the emitted gold answers 50/50 against the live GSM8K dataset. They match exactly,which matters because the verifier reward ([08_evaluation.md](08_evaluation.md)) is only as trustworthyas the gold it compares against.@@ -159,4 +159,4 @@ as the gold it compares against.| `arithmetic_prompts.jsonl` | `{prompt, gold}` | GRPO curriculum warm-up |<br>-➡️ Next: [Stage 1 — Pretraining](02_pretraining.md).+➡️ Next: [Stage 1: Pretraining](02_pretraining.md).diff --git a/docs/02_pretraining.md b/docs/02_pretraining.mdindex b8cd08a..7faa5b5 100644--- a/docs/02_pretraining.md+++ b/docs/02_pretraining.md@@ -1,12 +1,12 @@<!-- omit in toc -->-# Stage 1 — Pretraining the base model+# Stage 1: Pretraining the base modelEverything downstream is only as good as the base model, so the first thing I do is pretrain a**~400M-parameter** version of this repo's own `Transformer` from scratch on the Pile. The original[`train_transformer.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/train_transformer.py) is a clean single-GPU loop; for a mid-sizemodel on 2×H100 I wrote [`pretrain_base.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/pretrain_base.py), which adds the few things-that actually matter at this scale — DistributedDataParallel, bf16 autocast, gradient accumulation, a-cosine LR schedule with warmup, and periodic checkpointing — without touching the model itself.+that actually matter at this scale (DistributedDataParallel, bf16 autocast, gradient accumulation, a+cosine LR schedule with warmup, and periodic checkpointing) without touching the model itself.If the architecture or training terms are unfamiliar, read the foundations chapters first:[Decoder-Only Transformer](foundations/transformer.md),@@ -66,11 +66,11 @@ optimizer.step()A few choices worth calling out:- **bf16 autocast** ([`amp_autocast`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/utils.py)) needs no `GradScaler` (unlikefp16), which keeps the loop clean. Master weights stay fp32.-- **AdamW with a weight-decay split** ([`configure_optimizer`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/optim.py)) — decay+- **AdamW with a weight-decay split** ([`configure_optimizer`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/optim.py)): decaythe 2-D weight matrices, never the biases / norms / embeddings (the standard GPT recipe).-- **Cosine LR with warmup** ([`cosine_lr`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/optim.py)) — linear ramp for+- **Cosine LR with warmup** ([`cosine_lr`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/optim.py)): linear ramp for`warmup_steps`, then cosine decay to `min_lr`.-- **DDP** ([`distributed.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/distributed.py)) — each rank seeds its data shuffle+- **DDP** ([`distributed.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/distributed.py)): each rank seeds its data shuffledifferently so the two GPUs see different windows; only rank 0 logs and checkpoints.## Run it@@ -90,13 +90,13 @@ PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True PYTHONPATH=. \## What the numbers mean-- **train_loss** — running cross-entropy; starts near `ln(vocab) ≈ 10.8` and should fall steadily+- **train_loss**: running cross-entropy; starts near `ln(vocab) ≈ 10.8` and should fall steadily(mine went `11.06 → 8.6 → 6.0 → …`). This is the single best health signal.-- **tok/s** — throughput (~32k/s combined on 2×H100 here).-- **eval train/dev** — averaged loss on held-out windows, printed every `eval_steps`; watch the dev+- **tok/s**: throughput (~32k/s combined on 2×H100 here).+- **eval train/dev**: averaged loss on held-out windows, printed every `eval_steps`; watch the devloss to spot overfitting.Checkpoints are written to `/ephemeral/ckpts/base_pretrained.pt` every `save_every` steps and carry theconfig, so every later stage can rebuild the exact model with [`load_backbone_from_ckpt`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/utils.py).-➡️ Next: [Stage 2 — SFT](03_sft.md).+➡️ Next: [Stage 2: SFT](03_sft.md).diff --git a/docs/03_sft.md b/docs/03_sft.mdindex fb15a7a..bbb686d 100644--- a/docs/03_sft.md+++ b/docs/03_sft.md@@ -1,5 +1,5 @@<!-- omit in toc -->-# Stage 2 — Supervised Fine-Tuning (SFT)+# Stage 2: Supervised Fine-Tuning (SFT)The base model can continue text but it doesn't know it's supposed to *answer* you. SFT fixes that byshowing it thousands of `(instruction, response)` pairs and training it to produce the **response**.@@ -56,8 +56,8 @@ logits keeps the cross-entropy numerically clean under bf16.## The trainer[`train_sft.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/train_sft.py) loads the pretrained base with-[`load_backbone_from_ckpt`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/utils.py), then runs a compact loop — autocast forward,-masked loss, clip, step, cosine LR — with periodic dev evaluation:+[`load_backbone_from_ckpt`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/utils.py), then runs a compact loop (autocast forward,+masked loss, clip, step, cosine LR) with periodic dev evaluation:```pythontokens, mask, epoch = next(train_it)@@ -82,14 +82,14 @@ PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_sft.py # bot## What the numbers mean-- **train_loss / ppl** — masked cross-entropy (and its perplexity) over assistant tokens; should drop+- **train_loss / ppl**: masked cross-entropy (and its perplexity) over assistant tokens; should dropwell below the base model's loss. To sanity-check the mechanics I ran an *overfit* test on 8 rows andwatched the loss collapse `11.0 → 4.7`, confirming the gradient path learns.-- **dev_loss** — the same masked loss on a held-out split (`sft_dev_packed.h5`); the honest signal.-- **GSM8K dev accuracy** — after SFT the model both follows instructions *and* emits the+- **dev_loss**: the same masked loss on a held-out split (`sft_dev_packed.h5`); the honest signal.+- **GSM8K dev accuracy**: after SFT the model both follows instructions *and* emits the`<answer>…</answer>` format, so this should rise above the base model (see [08_evaluation.md](08_evaluation.md)).The result is saved to `/ephemeral/ckpts/sft.pt` and becomes the starting point for the reward model,DPO, PPO and GRPO.-➡️ Next: [Stage 3 — Reward Model](04_reward_model.md) or jump to [DPO](05_dpo.md).+➡️ Next: [Stage 3: Reward Model](04_reward_model.md) or jump to [DPO](05_dpo.md).diff --git a/docs/04_reward_model.md b/docs/04_reward_model.mdindex 892f2d5..db4969a 100644--- a/docs/04_reward_model.md+++ b/docs/04_reward_model.md@@ -1,9 +1,9 @@<!-- omit in toc -->-# Stage 3 — Reward Model+# Stage 3: Reward ModelTo do classic RLHF (PPO) we need something that scores a response with a single number: higher = morepreferred. That's the reward model. I build it by putting a tiny scalar head on top of the SFT-backbone and training it on human preference pairs with the **Bradley-Terry** loss — the same recipe+backbone and training it on human preference pairs with the **Bradley-Terry** loss, the same recipeas InstructGPT.This page assumes you already know how the backbone produces hidden states. If not, start with@@ -39,7 +39,7 @@ flowchart LR[`RewardModel`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_model.py#L37) wraps a `Transformer`, drops the `lm_head`,and reads the reward off the **last real token's** hidden state (the InstructGPT convention). Becauseattention is causal, that last token has seen the whole sequence and never attends to the right-padding-after it — so we need no attention mask:+after it, so we need no attention mask:```pythonclass RewardModel(nn.Module):@@ -64,8 +64,8 @@ def bradley_terry_loss(chosen_rewards, rejected_rewards):return -F.logsigmoid(chosen_rewards - rejected_rewards).mean()```-[`preference_accuracy`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_train.py#L23) — the fraction of pairs where-`r_chosen > r_rejected` — is the metric I actually watch.+[`preference_accuracy`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_train.py#L23), the fraction of pairs where+`r_chosen > r_rejected`, is the metric I actually watch.## The trainer@@ -94,12 +94,12 @@ PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_reward.py## What the numbers mean-- **loss** — Bradley-Terry; starts at `-log σ(0) = 0.693` (chance) and drops as the gap widens.-- **train_acc / test_acc** — preference accuracy. On clean fixtures it goes to `1.0`; on **real, noisy**- HH-RLHF / UltraFeedback expect roughly **0.65–0.75** — that's normal, human preferences are noisy.-- **margin** — mean `r_chosen − r_rejected`; a useful "is it still separating them" signal.+- **loss**: Bradley-Terry; starts at `-log σ(0) = 0.693` (chance) and drops as the gap widens.+- **train_acc / test_acc**: preference accuracy. On clean fixtures it goes to `1.0`; on **real, noisy**+ HH-RLHF / UltraFeedback expect roughly **0.65 to 0.75**. That's normal: human preferences are noisy.+- **margin**: mean `r_chosen − r_rejected`; a useful "is it still separating them" signal.Saved to `/ephemeral/ckpts/reward.pt`; PPO loads it with[`load_reward_model`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/reward_model.py) when `--reward_source rm`.-➡️ Next: [Stage 5 — PPO](06_ppo.md) (which consumes this), or the RM-free path: [DPO](05_dpo.md).+➡️ Next: [Stage 5: PPO](06_ppo.md) (which consumes this), or the RM-free path: [DPO](05_dpo.md).diff --git a/docs/05_dpo.md b/docs/05_dpo.mdindex 52b9425..cd7165e 100644--- a/docs/05_dpo.md+++ b/docs/05_dpo.md@@ -1,9 +1,9 @@<!-- omit in toc -->-# Stage 4 — DPO (and ORPO / KTO)+# Stage 4: DPO (and ORPO / KTO)Direct Preference Optimization is the shortcut around RLHF: instead of training a reward model and thenrunning an RL loop, DPO optimizes the policy *directly* on preference pairs, using a frozen copy of the-SFT model as a reference anchor. No reward model, no rollouts, no value function — just one clean loss.+SFT model as a reference anchor. No reward model, no rollouts, no value function: just one clean loss.I also implemented two popular variants behind a `--loss_type` flag: **ORPO** (reference-free) and**KTO** (works from a desirable/undesirable signal).@@ -64,9 +64,9 @@ def dpo_loss(policy_chosen_logps, policy_rejected_logps, ref_chosen_logps, ref_rThe two **variants** ([`orpo_loss`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/dpo.py#L48),[`kto_loss`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/dpo.py#L71)) live in the same file:-- **ORPO** — reference-free: combines the SFT negative-log-likelihood on the chosen response with an+- **ORPO**: reference-free: combines the SFT negative-log-likelihood on the chosen response with anodds-ratio preference term, folding SFT + alignment into one stage (no frozen reference needed).-- **KTO** — treats chosen as *desirable* and rejected as *undesirable* against a reference-KL baseline+- **KTO**: treats chosen as *desirable* and rejected as *undesirable* against a reference-KL baselineestimated from the batch; useful when you only have thumbs-up/down rather than pairs.## The trainer@@ -91,17 +91,17 @@ PYTHONPATH=. python scripts/train_dpo.py --loss_type orpo --orpo_lambda 1.0PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_dpo.py```-> DPO uses a **small** learning rate (`5e-7` by default) — it's easy to over-push away from the+> DPO uses a **small** learning rate (`5e-7` by default). It's easy to over-push away from the> reference and degrade the model, so go gentle.## What the numbers mean-- **loss** — DPO/KTO start near `0.693`; ORPO starts higher (it includes the NLL term).-- **acc** — implicit-reward accuracy: fraction of pairs where the policy's implicit reward prefers the+- **loss**: DPO/KTO start near `0.693`; ORPO starts higher (it includes the NLL term).+- **acc**: implicit-reward accuracy: fraction of pairs where the policy's implicit reward prefers thechosen response. Should climb above 0.5.-- **r_chosen / r_rejected** — the implicit rewards `β·(logπ − logref)`; the gap (margin) should widen.-- **GSM8K dev accuracy** — the real downstream check.+- **r_chosen / r_rejected**: the implicit rewards `β·(logπ − logref)`; the gap (margin) should widen.+- **GSM8K dev accuracy**: the real downstream check.Saved to `/ephemeral/ckpts/dpo.pt`.-➡️ Next: the RL path — [PPO](06_ppo.md) and [GRPO](07_grpo.md).+➡️ Next: the RL path, [PPO](06_ppo.md) and [GRPO](07_grpo.md).diff --git a/docs/06_ppo.md b/docs/06_ppo.mdindex 103c74c..e215a1a 100644--- a/docs/06_ppo.md+++ b/docs/06_ppo.md@@ -1,8 +1,8 @@<!-- omit in toc -->-# Stage 5 — PPO (classic RLHF)+# Stage 5: PPO (classic RLHF)This is the original ChatGPT recipe: let the model generate, score the generations with a reward, and-nudge the policy toward higher-reward behaviour using Proximal Policy Optimization — with a value+nudge the policy toward higher-reward behaviour using Proximal Policy Optimization, with a valuenetwork (critic) for variance reduction and a KL penalty to keep it from drifting too far from the SFTmodel. I wrote the whole loop from scratch: rollout → reward → GAE advantages → clipped update.@@ -39,7 +39,7 @@ flowchart LR## The actor-criticPPO needs a per-token value estimate `V(s_t)` next to the policy logits. I get both from one backbone-with [`TransformerWithValueHead`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/value_head.py#L19) — it reuses+with [`TransformerWithValueHead`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/value_head.py#L19). It reuses`forward_hidden` + `lm_head` for the policy and adds a small scalar value head (initialized to ~0 so thecritic doesn't destabilize early training):@@ -58,7 +58,7 @@ completion for each, and [`generate_with_logprobs`](https://github.com/FareedKhasampling log-probs. Log-probs are always taken in **fp32** ([`compute_logprobs`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rollout.py#L233))because PPO subtracts them and bf16 rounding there is harmful.-## GAE — Generalized Advantage Estimation+## GAE: Generalized Advantage Estimation[`compute_gae`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/ppo.py#L24) works in the "action frame" (index `t` = producingtoken `t+1`), bootstrapping only while the next action is still a response token:@@ -101,16 +101,16 @@ PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_ppo.py## What the numbers mean-- **reward** — mean task reward per iteration; the headline curve, should trend up.-- **KL_ref** — mean KL of the policy from the SFT reference; must stay **bounded**. If it blows up the- model is degenerating — lower the LR or raise `--kl_coef`.-- **clipfrac** — fraction of tokens hitting the PPO clip; a health/▒step-size signal.-- **value_loss** — critic regression error.-- **GSM8K test accuracy** — the real outcome, evaluated every `--eval_every`.+- **reward**: mean task reward per iteration; the headline curve, should trend up.+- **KL_ref**: mean KL of the policy from the SFT reference; must stay **bounded**. If it blows up the+ model is degenerating: lower the LR or raise `--kl_coef`.+- **clipfrac**: fraction of tokens hitting the PPO clip; a health and step-size signal.+- **value_loss**: critic regression error.+- **GSM8K test accuracy**: the real outcome, evaluated every `--eval_every`.> PPO is the touchy one: small LR (`1e-6`), `clip 0.2`, grad-clip 1.0, and watch KL. I verified the loop-> truly *optimizes* by giving it a learnable synthetic reward — reward climbed `0.10 → 1.00`.+> truly *optimizes* by giving it a learnable synthetic reward: reward climbed `0.10 → 1.00`.Saved to `/ephemeral/ckpts/ppo.pt`.-➡️ Next: [Stage 6 — GRPO](07_grpo.md), which drops the critic entirely.+➡️ Next: [Stage 6: GRPO](07_grpo.md), which drops the critic entirely.diff --git a/docs/07_grpo.md b/docs/07_grpo.mdindex 796aa5a..a0b9d46 100644--- a/docs/07_grpo.md+++ b/docs/07_grpo.md@@ -1,10 +1,10 @@<!-- omit in toc -->-# Stage 6 — GRPO / RLVR (the reasoning frontier)+# Stage 6: GRPO / RLVR (the reasoning frontier)GRPO (Group Relative Policy Optimization) is the algorithm behind DeepSeek-R1, and it's beautifullysimple: **throw away PPO's value network**. For each prompt, sample a whole *group* of answers, scorethem with a verifiable reward, and use the group's own mean/std as the baseline. The advantage is just-"how much better than your groupmates was this answer?" — no critic to train, no value loss.+"how much better than your groupmates was this answer?" No critic to train, no value loss.For the group-relative advantage formula and how it relates to PPO-style policy ratios, see[Objectives, Losses & Perplexity](foundations/objectives.md).@@ -36,7 +36,7 @@ flowchart LR## Group-relative advantage-[`group_advantages`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/grpo.py#L17) is the whole idea — standardize rewards *within+[`group_advantages`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/grpo.py#L17) is the whole idea: standardize rewards *withineach group*, so a good answer is one that beats its siblings on the same prompt:```python@@ -47,7 +47,7 @@ def group_advantages(rewards, group_size, eps=1e-4):```A nice property: if every answer in a group gets the same reward (all right or all wrong), the std-based-advantage is ~0 and that group simply contributes no gradient — so I log the fraction of *informative*+advantage is ~0 and that group simply contributes no gradient, so I log the fraction of *informative*groups as a health metric.## The loss: clipped surrogate + KL@@ -87,11 +87,11 @@ PYTHONPATH=. torchrun --standalone --nproc_per_node=2 scripts/train_grpo.py## What the numbers mean-- **reward** — mean verifier reward across the group samples; the curve you want climbing.-- **informative** — fraction of groups with non-zero reward spread (groups that actually teach+- **reward**: mean verifier reward across the group samples; the curve you want climbing.+- **informative**: fraction of groups with non-zero reward spread (groups that actually teachsomething). If this collapses to 0, raise temperature / group size or stay longer on the curriculum.-- **KL** — KL to the reference; keep it bounded.-- **GSM8K test accuracy** — the headline reasoning metric, evaluated every `--eval_every`.+- **KL**: KL to the reference; keep it bounded.+- **GSM8K test accuracy**: the headline reasoning metric, evaluated every `--eval_every`.> I verified the GRPO path genuinely optimizes: with a learnable reward the mean reward climbed> **0.10 → 0.69 → 1.00** in ~15 iterations and saturated. PPO and GRPO share the same rollout/log-probdiff --git a/docs/08_evaluation.md b/docs/08_evaluation.mdindex a5c2941..8c2d1f2 100644--- a/docs/08_evaluation.md+++ b/docs/08_evaluation.md@@ -3,8 +3,8 @@A pipeline is only believable if you can measure it, so I evaluate every stage on the **same** held-outGSM8K test set with greedy decoding. The headline deliverable is a single table: GSM8K accuracy as it-moves Base → SFT → DPO → PPO → GRPO. The reward is *verifiable* — I parse the model's final number and-compare it to the gold answer — so the score is objective, not a judgment call.+moves Base → SFT → DPO → PPO → GRPO. The reward is *verifiable* (I parse the model's final number and+compare it to the gold answer), so the score is objective, not a judgment call.@@ -44,9 +44,9 @@ responses = batched_generate(model, prompts, max_new_tokens, device=device, greecorrect = sum(is_correct(resp, gsm8k_gold_answer(ans)) for (q, ans), resp in zip(qa_pairs, responses))```-The reward/checker lives in [`rewards/`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rewards/). `extract_answer` is tolerant —+The reward/checker lives in [`rewards/`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rewards/). `extract_answer` is tolerant:it prefers an `<answer>…</answer>` tag, then a GSM8K-style `#### N`, then falls back to the last number-in the text — and [`reward_gsm8k`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rewards/verifiers.py#L35) is+in the text. [`reward_gsm8k`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/rewards/verifiers.py#L35) is**correctness-dominant** with only a small, bounded format bonus, to discourage reward hacking:```python@@ -85,14 +85,14 @@ grpo ... 200## In-training metricsEach trainer also writes a metrics JSONL under `/ephemeral/logs/` (via-[`MetricsLogger`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/logging_utils.py)) — train/dev loss for SFT, preference accuracy+[`MetricsLogger`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/logging_utils.py)): train/dev loss for SFT, preference accuracyfor the reward model, implicit-reward accuracy for DPO, and reward/KL/clip-fraction + GSM8K accuracy forPPO/GRPO. Pass `--use_wandb true` to also mirror to Weights & Biases; the JSONL is always written so youcan plot offline.## What "good" looks like at this scale-A ~400M from-scratch model won't top the GSM8K leaderboard — the point is the **relative** climb across+A ~400M from-scratch model won't top the GSM8K leaderboard. The point is the **relative** climb acrossstages and bounded KL during RL. Expect modest absolute numbers but a clear, real before/after gain ateach step.diff --git a/docs/09_inference.md b/docs/09_inference.mdindex 8e9df26..c566cdf 100644--- a/docs/09_inference.md+++ b/docs/09_inference.md@@ -1,9 +1,9 @@<!-- omit in toc --># Inference & Chat-Training is only satisfying if you can actually *talk* to the result. The original-[`generate_text.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/generate_text.py) does raw continuation for the base model, but it's-hard-wired to the legacy config and has no chat template — so I added a small inference layer that loads+Training is only satisfying if you can actually *talk* to the result.+[`generate_text.py`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/scripts/generate_text.py) continues text with the models from `train_transformer.py`, but it+has no chat template, so I added a small inference layer that loads**any** stage checkpoint (base / SFT / DPO / PPO / GRPO) and talks to it correctly.For the underlying decoding loop, context cropping, temperature, and stop-token behavior, read@@ -49,9 +49,9 @@ state = {k.removeprefix("module.").removeprefix("transformer."): v for k, v in s[`generate_reply`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/inference.py#L37) has two modes, reusing the same testedgeneration core as training/eval ([`batched_generate`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/evaluation.py#L24)):-- **chat** (default) — wraps your text in the chat template (optionally with a `system` message) and+- **chat** (default): wraps your text in the chat template (optionally with a `system` message) andreturns the decoded assistant turn. Use this for SFT/DPO/PPO/GRPO checkpoints.-- **raw** (`--raw`) — treats your text as a prefix and returns the base model's continuation (no+- **raw** (`--raw`): treats your text as a prefix and returns the base model's continuation (notemplate). Use this for `base_pretrained.pt`.```python@@ -63,7 +63,7 @@ out = batched_generate(model, [ids], max_new_tokens, device=device,temperature=temperature, top_k=top_k, top_p=top_p, greedy=greedy)```-Decoding is defensive — [`decode`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/chat_template.py) drops the EOT terminator and+Decoding is defensive: [`decode`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/post_training/chat_template.py) drops the EOT terminator andany padding-vocab ids (the model's vocab is padded to 50304 but r50k_base only decodes 0–50255).## The CLI@@ -85,8 +85,8 @@ on `--device cuda` or `cpu` (both verified).## Sampling knobs, briefly-- **greedy** — reproducible, best for eval / math (`--greedy`).-- **temperature** — higher = more random; ~`0.7–1.0` for open-ended chat.-- **top_p / top_k** — nucleus / top-k truncation to cut the long tail of unlikely tokens.+- **greedy**: reproducible, best for eval / math (`--greedy`).+- **temperature**: higher = more random; ~`0.7–1.0` for open-ended chat.+- **top_p / top_k**: nucleus / top-k truncation to cut the long tail of unlikely tokens.That's the full loop: pretrain → align → reason → measure → chat. Back to the [overview](README.md).diff --git a/docs/README.md b/docs/README.mdindex 04db457..8f872b5 100644--- a/docs/README.md+++ b/docs/README.md@@ -1,9 +1,9 @@<!-- omit in toc -->-# Post-Training & Alignment — Overview+# Post-Training & Alignment: OverviewWhen I first trained this transformer from scratch, it could *continue* text but it couldn't*follow instructions* or *reason*. That's what post-training fixes. This `docs/` folder walks-through the whole journey I built on top of the base model — every stage written from scratch+through the whole journey I built on top of the base model, every stage written from scratchin plain PyTorch (no `trl`, no `peft`, no `transformers`), trained on real public datasets, andrunnable on a single GPU or scaled across multiple GPUs with DDP.@@ -74,14 +74,14 @@ flowchart TD| 4 | **DPO / ORPO / KTO** | to prefer better answers *without* an RL loop | [05_dpo.md](05_dpo.md) || 5 | **PPO** | to maximize a reward (RM or verifier) with the classic RLHF loop | [06_ppo.md](06_ppo.md) || 6 | **GRPO / RLVR** | to reason, using verifiable rewards (DeepSeek-R1 style) | [07_grpo.md](07_grpo.md) |-| — | **Data pipeline** | how every dataset above is downloaded & preprocessed | [01_data_pipeline.md](01_data_pipeline.md) |-| — | **Evaluation** | how I measure GSM8K accuracy across all stages | [08_evaluation.md](08_evaluation.md) |-| — | **Inference / chat** | how to actually talk to any checkpoint | [09_inference.md](09_inference.md) |+| | **Data pipeline** | how every dataset above is downloaded & preprocessed | [01_data_pipeline.md](01_data_pipeline.md) |+| | **Evaluation** | how I measure GSM8K accuracy across all stages | [08_evaluation.md](08_evaluation.md) |+| | **Inference / chat** | how to actually talk to any checkpoint | [09_inference.md](09_inference.md) |## The one design rule: *wrap, don't rewrite*Everything here sits on top of the original [`Transformer`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/models/transformer.py). I changed the-educational model in exactly **one** place — I added a [`forward_hidden`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/models/transformer.py#L56)+educational model in exactly **one** place: I added a [`forward_hidden`](https://github.com/FareedKhan-dev/train-llm-from-scratch/blob/main/src/models/transformer.py#L56)method that returns the final hidden states the `lm_head` consumes. Every post-training head (a valuehead for PPO, a scalar reward head for the reward model) and every RL log-prob computation composes*around* that one method, so the from-scratch model you already understand stays intact.@@ -92,8 +92,8 @@ head for PPO, a scalar reward head for the reward model) and every RL log-prob c· 🟧 RL / reward · 🟥 loss / objective · 🟪 evaluation · ⬜ checkpoint> Each diagram is a hand-drawn, colour-coded Mermaid sketch, **pre-rendered to a PNG and embedded as-> an image** (GitHub's live Mermaid doesn't reliably do `look: handDrawn`, and some viewers — e.g. the-> VS Code preview — block SVGs, so an embedded PNG shows everywhere). The editable Mermaid source sits+> an image** (GitHub's live Mermaid doesn't reliably do `look: handDrawn`, and some viewers, e.g. the+> VS Code preview, block SVGs, so an embedded PNG shows everywhere). The editable Mermaid source sits> in a collapsible *"Mermaid source"* block under each image. To regenerate the images after editing,> see [diagrams/README.md](diagrams/README.md).