Skip to content

A straightforward method for training your LLM, from downloading data to generating text.

docs(diagrams): GRPO variants, the laptop student track and LoRA

FareedKhan-devc529fbc8fed6
diff --git a/docs/diagrams/20_grpo_variants.png b/docs/diagrams/20_grpo_variants.png
new file mode 100644
index 0000000..abcf246
Binary files /dev/null and b/docs/diagrams/20_grpo_variants.png differ
diff --git a/docs/diagrams/21_student_track.png b/docs/diagrams/21_student_track.png
new file mode 100644
index 0000000..079a8ae
Binary files /dev/null and b/docs/diagrams/21_student_track.png differ
diff --git a/docs/diagrams/22_lora.png b/docs/diagrams/22_lora.png
new file mode 100644
index 0000000..3645fe5
Binary files /dev/null and b/docs/diagrams/22_lora.png differ
diff --git a/docs/diagrams/src/20_grpo_variants.mmd b/docs/diagrams/src/20_grpo_variants.mmd
new file mode 100644
index 0000000..0079106
--- /dev/null
+++ b/docs/diagrams/src/20_grpo_variants.mmd
@@ -0,0 +1,18 @@
+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%
+flowchart LR
+ P([prompt]):::data --> G{{sample a group<br/>of G answers}}:::model
+ G --> V[verifier reward<br/>per answer]:::rl
+ V --> F{same reward<br/>for all G?}:::rl
+ F -->|yes, DAPO: skip the group| SKIP[masked out<br/>of the loss]:::ckpt
+ F -->|no| A[advantage = r - mean<br/>GRPO also divides by std<br/>Dr. GRPO does not]:::proc
+ A --> R[importance ratio<br/>per token: GRPO, DAPO<br/>per answer: GSPO]:::proc
+ R --> C[clip the ratio<br/>DAPO: wider upper bound]:::proc
+ C --> L[average the token terms<br/>per answer: GRPO<br/>over all tokens: DAPO<br/>sum / budget: Dr. GRPO]:::loss
+ L --> U{{policy update}}:::model
+ U -. next prompts .-> P
+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;
+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;
+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;
+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;
+ classDef loss fill:#ffd6d6,stroke:#c0392b,stroke-width:2px,color:#5c1212;
+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;
diff --git a/docs/diagrams/src/21_student_track.mmd b/docs/diagrams/src/21_student_track.mmd
new file mode 100644
index 0000000..d393743
--- /dev/null
+++ b/docs/diagrams/src/21_student_track.mmd
@@ -0,0 +1,13 @@
+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%
+flowchart LR
+ D([TinyStories<br/>or your own text]):::data --> BPE[train a BPE tokenizer<br/>from scratch, 4096 ids]:::proc
+ BPE --> H[(data/tiny<br/>train.h5, val.h5)]:::store
+ H --> T{{train_transformer.py<br/>--preset tiny / student}}:::model
+ T --> C[(models/tiny.pt<br/>weights + tokenizer)]:::ckpt
+ C --> G([generate_text.py<br/>stories on your laptop]):::eval
+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;
+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;
+ classDef store fill:#cdece8,stroke:#16a085,stroke-width:2px,color:#0a3d33;
+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;
+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;
+ classDef eval fill:#e8d6ff,stroke:#8e44ad,stroke-width:2px,color:#3d1a5a;
diff --git a/docs/diagrams/src/22_lora.mmd b/docs/diagrams/src/22_lora.mmd
new file mode 100644
index 0000000..d2b2fe4
--- /dev/null
+++ b/docs/diagrams/src/22_lora.mmd
@@ -0,0 +1,15 @@
+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%
+flowchart LR
+ X([x, size d_in]):::data --> W[frozen weight W<br/>d_out x d_in]:::ckpt
+ X --> A[A: r x d_in<br/>trainable, random init]:::rl
+ A --> B[B: d_out x r<br/>trainable, starts at 0]:::rl
+ B --> S[times alpha / r]:::proc
+ W --> SUM((+)):::proc
+ S --> SUM
+ SUM --> Y([y]):::data
+ SUM -. after training .-> M[merge: W + alpha/r x B A<br/>one plain matrix again]:::model
+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;
+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;
+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;
+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;
+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;