A straightforward method for training your LLM, from downloading data to generating text.
docs(diagrams): the modern decoder block, GQA, the KV cache and MoE
FareedKhan-devc71fc7867189
diff --git a/docs/diagrams/16_modern_block.png b/docs/diagrams/16_modern_block.pngnew file mode 100644index 0000000..e85caecBinary files /dev/null and b/docs/diagrams/16_modern_block.png differdiff --git a/docs/diagrams/17_gqa.png b/docs/diagrams/17_gqa.pngnew file mode 100644index 0000000..722d00eBinary files /dev/null and b/docs/diagrams/17_gqa.png differdiff --git a/docs/diagrams/18_kv_cache.png b/docs/diagrams/18_kv_cache.pngnew file mode 100644index 0000000..7a83e00Binary files /dev/null and b/docs/diagrams/18_kv_cache.png differdiff --git a/docs/diagrams/19_moe.png b/docs/diagrams/19_moe.pngnew file mode 100644index 0000000..2a3471fBinary files /dev/null and b/docs/diagrams/19_moe.png differdiff --git a/docs/diagrams/src/16_modern_block.mmd b/docs/diagrams/src/16_modern_block.mmdnew file mode 100644index 0000000..78fb290--- /dev/null+++ b/docs/diagrams/src/16_modern_block.mmd@@ -0,0 +1,20 @@+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%+flowchart LR+ X[(residual stream x)]:::store --> N1[RMSNorm]:::proc+ N1 --> QKV[q, k, v projections<br/>fewer k, v heads: GQA]:::proc+ QKV --> QK[QK-norm + RoPE<br/>rotate q and k by position]:::rl+ QK --> SDPA{{scaled_dot_product_attention<br/>causal or sliding window}}:::model+ KV[(KV cache<br/>during generation)]:::ckpt -. past keys and values .-> SDPA+ SDPA --> O[output projection]:::proc+ O --> A1((+)):::proc+ X --> A1+ A1 --> N2[RMSNorm]:::proc+ N2 --> MLP{{SwiGLU<br/>or Mixture of Experts}}:::model+ MLP --> A2((+)):::proc+ A1 --> A2+ A2 --> OUT[(next block)]:::store+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;+ classDef store fill:#cdece8,stroke:#16a085,stroke-width:2px,color:#0a3d33;+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;diff --git a/docs/diagrams/src/17_gqa.mmd b/docs/diagrams/src/17_gqa.mmdnew file mode 100644index 0000000..33d13a1--- /dev/null+++ b/docs/diagrams/src/17_gqa.mmd@@ -0,0 +1,30 @@+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%+flowchart LR+ subgraph MHA[multi-head: 4 key/value heads]+ direction TB+ x1([x]):::data --> a1[q]:::proc & a2[q]:::proc & a3[q]:::proc & a4[q]:::proc+ a1 --> b1[(k, v)]:::store+ a2 --> b2[(k, v)]:::store+ a3 --> b3[(k, v)]:::store+ a4 --> b4[(k, v)]:::store+ end+ subgraph GQA[grouped-query: 2 key/value heads]+ direction TB+ x2([x]):::data --> c1[q]:::proc & c2[q]:::proc & c3[q]:::proc & c4[q]:::proc+ c1 --> d1[(k, v)]:::store+ c2 --> d1+ c3 --> d2[(k, v)]:::store+ c4 --> d2+ end+ subgraph MQA[multi-query: 1 key/value head]+ direction TB+ x3([x]):::data --> e1[q]:::proc & e2[q]:::proc & e3[q]:::proc & e4[q]:::proc+ e1 --> f1[(k, v)]:::store+ e2 --> f1+ e3 --> f1+ e4 --> f1+ end+ MHA ~~~ GQA ~~~ MQA+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;+ classDef store fill:#cdece8,stroke:#16a085,stroke-width:2px,color:#0a3d33;diff --git a/docs/diagrams/src/18_kv_cache.mmd b/docs/diagrams/src/18_kv_cache.mmdnew file mode 100644index 0000000..ba68bcd--- /dev/null+++ b/docs/diagrams/src/18_kv_cache.mmd@@ -0,0 +1,14 @@+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%+flowchart LR+ P([prompt: T tokens]):::data --> PF{{prefill<br/>one pass over the prompt}}:::model+ PF -->|store k, v of every layer| C[(KV cache<br/>T tokens)]:::ckpt+ PF --> S1[sample token T+1]:::rl+ S1 --> D{{decode<br/>run ONLY the new token}}:::model+ C -->|attend to the cached k, v| D+ D -->|append its k, v| C+ D --> S2[sample the next token]:::rl+ S2 -->|repeat| D+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;diff --git a/docs/diagrams/src/19_moe.mmd b/docs/diagrams/src/19_moe.mmdnew file mode 100644index 0000000..0aca89e--- /dev/null+++ b/docs/diagrams/src/19_moe.mmd@@ -0,0 +1,18 @@+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%+flowchart LR+ T([token vector x]):::data --> R{router<br/>softmax over experts}:::rl+ R -->|top-k = 2| E1{{expert 3}}:::model+ R -->|top-k = 2| E2{{expert 7}}:::model+ R -. not chosen .-> E0[experts 1, 2, 4, 5, 6, 8<br/>idle for this token]:::ckpt+ T --> SH{{shared expert<br/>every token}}:::model+ E1 -->|weight p3| SUM((+)):::proc+ E2 -->|weight p7| SUM+ SH --> SUM+ SUM --> Y([output]):::data+ R -. how evenly were tokens spread? .-> AUX[load-balancing loss<br/>E x sum of f_e x P_e]:::loss+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;+ classDef loss fill:#ffd6d6,stroke:#c0392b,stroke-width:2px,color:#5c1212;+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;