A straightforward method for training your LLM, from downloading data to generating text.
docs(diagrams): GRPO variants, the laptop student track and LoRA
FareedKhan-devc529fbc8fed6
diff --git a/docs/diagrams/20_grpo_variants.png b/docs/diagrams/20_grpo_variants.pngnew file mode 100644index 0000000..abcf246Binary files /dev/null and b/docs/diagrams/20_grpo_variants.png differdiff --git a/docs/diagrams/21_student_track.png b/docs/diagrams/21_student_track.pngnew file mode 100644index 0000000..079a8aeBinary files /dev/null and b/docs/diagrams/21_student_track.png differdiff --git a/docs/diagrams/22_lora.png b/docs/diagrams/22_lora.pngnew file mode 100644index 0000000..3645fe5Binary files /dev/null and b/docs/diagrams/22_lora.png differdiff --git a/docs/diagrams/src/20_grpo_variants.mmd b/docs/diagrams/src/20_grpo_variants.mmdnew file mode 100644index 0000000..0079106--- /dev/null+++ b/docs/diagrams/src/20_grpo_variants.mmd@@ -0,0 +1,18 @@+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%+flowchart LR+ P([prompt]):::data --> G{{sample a group<br/>of G answers}}:::model+ G --> V[verifier reward<br/>per answer]:::rl+ V --> F{same reward<br/>for all G?}:::rl+ F -->|yes, DAPO: skip the group| SKIP[masked out<br/>of the loss]:::ckpt+ F -->|no| A[advantage = r - mean<br/>GRPO also divides by std<br/>Dr. GRPO does not]:::proc+ A --> R[importance ratio<br/>per token: GRPO, DAPO<br/>per answer: GSPO]:::proc+ R --> C[clip the ratio<br/>DAPO: wider upper bound]:::proc+ C --> L[average the token terms<br/>per answer: GRPO<br/>over all tokens: DAPO<br/>sum / budget: Dr. GRPO]:::loss+ L --> U{{policy update}}:::model+ U -. next prompts .-> P+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;+ classDef loss fill:#ffd6d6,stroke:#c0392b,stroke-width:2px,color:#5c1212;+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;diff --git a/docs/diagrams/src/21_student_track.mmd b/docs/diagrams/src/21_student_track.mmdnew file mode 100644index 0000000..d393743--- /dev/null+++ b/docs/diagrams/src/21_student_track.mmd@@ -0,0 +1,13 @@+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%+flowchart LR+ D([TinyStories<br/>or your own text]):::data --> BPE[train a BPE tokenizer<br/>from scratch, 4096 ids]:::proc+ BPE --> H[(data/tiny<br/>train.h5, val.h5)]:::store+ H --> T{{train_transformer.py<br/>--preset tiny / student}}:::model+ T --> C[(models/tiny.pt<br/>weights + tokenizer)]:::ckpt+ C --> G([generate_text.py<br/>stories on your laptop]):::eval+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;+ classDef store fill:#cdece8,stroke:#16a085,stroke-width:2px,color:#0a3d33;+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;+ classDef eval fill:#e8d6ff,stroke:#8e44ad,stroke-width:2px,color:#3d1a5a;diff --git a/docs/diagrams/src/22_lora.mmd b/docs/diagrams/src/22_lora.mmdnew file mode 100644index 0000000..d2b2fe4--- /dev/null+++ b/docs/diagrams/src/22_lora.mmd@@ -0,0 +1,15 @@+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%+flowchart LR+ X([x, size d_in]):::data --> W[frozen weight W<br/>d_out x d_in]:::ckpt+ X --> A[A: r x d_in<br/>trainable, random init]:::rl+ A --> B[B: d_out x r<br/>trainable, starts at 0]:::rl+ B --> S[times alpha / r]:::proc+ W --> SUM((+)):::proc+ S --> SUM+ SUM --> Y([y]):::data+ SUM -. after training .-> M[merge: W + alpha/r x B A<br/>one plain matrix again]:::model+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;