Skip to content

A straightforward method for training your LLM, from downloading data to generating text.

docs: chat's --int8 and automatic device, the modern pipeline and the full test suite in POST_TRAINING.md

FareedKhan-dev4e73b171c74f
diff --git a/POST_TRAINING.md b/POST_TRAINING.md
index c83b6a1..ad1a8d2 100644
--- a/POST_TRAINING.md
+++ b/POST_TRAINING.md
@@ -183,13 +183,19 @@ any external service); pass `--use_wandb true` to also mirror to Weights & Biase
enforce `prompt + generation ≤ context_length`.
- **Reward hacking / KL control.** Verifier rewards are correctness-dominant with a small,
bounded format bonus; a KL-to-reference penalty anchors RL to the SFT policy.
+- **Two architectures, one pipeline.** `"arch": "modern"` in `configs/base.json` switches every
+ stage to the modern decoder (RoPE, RMSNorm, SwiGLU, GQA or MLA, optional MoE); see
+ [docs/modern](docs/modern/README.md). Its rollouts, evaluation and chat decode with a KV cache.
## Tests
```bash
python tests/test_post_training_smoke.py # core math: log-probs, heads, parsing, masking
+pytest # the whole suite, CPU only
+pytest tests/test_pipeline_e2e.py -s # every training script, end to end, on generated data
```
-Each trainer also runs end-to-end on a tiny model in seconds (see the smoke commands used
-during development), and the PPO/GRPO math has standalone unit checks (GAE, clipped losses,
-group advantages, k3 KL).
+The end-to-end test writes a small generated dataset, then runs pretraining, SFT (with LoRA),
+the reward model, DPO and SimPO, PPO, GRPO and chat for a few steps each, with both
+architectures, so a change that breaks a stage or the hand-over between two stages fails in CI. The PPO/GRPO math has standalone unit checks (GAE, clipped losses, group
+advantages, k3 KL, and the Dr. GRPO, DAPO and GSPO variants).
diff --git a/docs/09_inference.md b/docs/09_inference.md
index a1d26fb..41b3417 100644
--- a/docs/09_inference.md
+++ b/docs/09_inference.md
@@ -80,8 +80,10 @@ python scripts/chat.py --ckpt models/base_pretrained.pt --raw --prompt "Once upo
python scripts/chat.py --ckpt models/sft.pt
```
-Sampling controls: `--temperature`, `--top_p`, `--top_k`, or `--greedy` for deterministic argmax. Runs
-on `--device cuda` or `cpu` (both verified).
+Sampling controls: `--temperature`, `--top_p`, `--top_k`, or `--greedy` for deterministic argmax.
+`--device` defaults to `auto` (CUDA, then Apple MPS, then the CPU), and `--int8` stores the linear
+layers' weights as int8, which takes about a quarter of their memory. Modern-architecture
+checkpoints generate with a KV cache, so each new token costs one short forward pass.
## Sampling knobs, briefly
@@ -89,4 +91,7 @@ on `--device cuda` or `cpu` (both verified).
- **temperature**: higher = more random; ~`0.7–1.0` for open-ended chat.
- **top_p / top_k**: nucleus / top-k truncation to cut the long tail of unlikely tokens.
+[Inference](modern/inference.md) in the Modern LLM section goes further: min-p sampling, how
+the KV cache works, speculative decoding and int8 weights, with measurements.
+
That's the full loop: pretrain → align → reason → measure → chat. Back to the [overview](README.md).