Skip to content

A straightforward method for training your LLM, from downloading data to generating text.

docs(diagrams): the modern decoder block, GQA, the KV cache and MoE

FareedKhan-devc71fc7867189
diff --git a/docs/diagrams/16_modern_block.png b/docs/diagrams/16_modern_block.png
new file mode 100644
index 0000000..e85caec
Binary files /dev/null and b/docs/diagrams/16_modern_block.png differ
diff --git a/docs/diagrams/17_gqa.png b/docs/diagrams/17_gqa.png
new file mode 100644
index 0000000..722d00e
Binary files /dev/null and b/docs/diagrams/17_gqa.png differ
diff --git a/docs/diagrams/18_kv_cache.png b/docs/diagrams/18_kv_cache.png
new file mode 100644
index 0000000..7a83e00
Binary files /dev/null and b/docs/diagrams/18_kv_cache.png differ
diff --git a/docs/diagrams/19_moe.png b/docs/diagrams/19_moe.png
new file mode 100644
index 0000000..2a3471f
Binary files /dev/null and b/docs/diagrams/19_moe.png differ
diff --git a/docs/diagrams/src/16_modern_block.mmd b/docs/diagrams/src/16_modern_block.mmd
new file mode 100644
index 0000000..78fb290
--- /dev/null
+++ b/docs/diagrams/src/16_modern_block.mmd
@@ -0,0 +1,20 @@
+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%
+flowchart LR
+ X[(residual stream x)]:::store --> N1[RMSNorm]:::proc
+ N1 --> QKV[q, k, v projections<br/>fewer k, v heads: GQA]:::proc
+ QKV --> QK[QK-norm + RoPE<br/>rotate q and k by position]:::rl
+ QK --> SDPA{{scaled_dot_product_attention<br/>causal or sliding window}}:::model
+ KV[(KV cache<br/>during generation)]:::ckpt -. past keys and values .-> SDPA
+ SDPA --> O[output projection]:::proc
+ O --> A1((+)):::proc
+ X --> A1
+ A1 --> N2[RMSNorm]:::proc
+ N2 --> MLP{{SwiGLU<br/>or Mixture of Experts}}:::model
+ MLP --> A2((+)):::proc
+ A1 --> A2
+ A2 --> OUT[(next block)]:::store
+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;
+ classDef store fill:#cdece8,stroke:#16a085,stroke-width:2px,color:#0a3d33;
+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;
+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;
+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;
diff --git a/docs/diagrams/src/17_gqa.mmd b/docs/diagrams/src/17_gqa.mmd
new file mode 100644
index 0000000..33d13a1
--- /dev/null
+++ b/docs/diagrams/src/17_gqa.mmd
@@ -0,0 +1,30 @@
+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%
+flowchart LR
+ subgraph MHA[multi-head: 4 key/value heads]
+ direction TB
+ x1([x]):::data --> a1[q]:::proc & a2[q]:::proc & a3[q]:::proc & a4[q]:::proc
+ a1 --> b1[(k, v)]:::store
+ a2 --> b2[(k, v)]:::store
+ a3 --> b3[(k, v)]:::store
+ a4 --> b4[(k, v)]:::store
+ end
+ subgraph GQA[grouped-query: 2 key/value heads]
+ direction TB
+ x2([x]):::data --> c1[q]:::proc & c2[q]:::proc & c3[q]:::proc & c4[q]:::proc
+ c1 --> d1[(k, v)]:::store
+ c2 --> d1
+ c3 --> d2[(k, v)]:::store
+ c4 --> d2
+ end
+ subgraph MQA[multi-query: 1 key/value head]
+ direction TB
+ x3([x]):::data --> e1[q]:::proc & e2[q]:::proc & e3[q]:::proc & e4[q]:::proc
+ e1 --> f1[(k, v)]:::store
+ e2 --> f1
+ e3 --> f1
+ e4 --> f1
+ end
+ MHA ~~~ GQA ~~~ MQA
+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;
+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;
+ classDef store fill:#cdece8,stroke:#16a085,stroke-width:2px,color:#0a3d33;
diff --git a/docs/diagrams/src/18_kv_cache.mmd b/docs/diagrams/src/18_kv_cache.mmd
new file mode 100644
index 0000000..ba68bcd
--- /dev/null
+++ b/docs/diagrams/src/18_kv_cache.mmd
@@ -0,0 +1,14 @@
+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%
+flowchart LR
+ P([prompt: T tokens]):::data --> PF{{prefill<br/>one pass over the prompt}}:::model
+ PF -->|store k, v of every layer| C[(KV cache<br/>T tokens)]:::ckpt
+ PF --> S1[sample token T+1]:::rl
+ S1 --> D{{decode<br/>run ONLY the new token}}:::model
+ C -->|attend to the cached k, v| D
+ D -->|append its k, v| C
+ D --> S2[sample the next token]:::rl
+ S2 -->|repeat| D
+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;
+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;
+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;
+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;
diff --git a/docs/diagrams/src/19_moe.mmd b/docs/diagrams/src/19_moe.mmd
new file mode 100644
index 0000000..0aca89e
--- /dev/null
+++ b/docs/diagrams/src/19_moe.mmd
@@ -0,0 +1,18 @@
+%%{init: {'look':'handDrawn','theme':'base','themeVariables':{'fontFamily':'Comic Sans MS, cursive'}}}%%
+flowchart LR
+ T([token vector x]):::data --> R{router<br/>softmax over experts}:::rl
+ R -->|top-k = 2| E1{{expert 3}}:::model
+ R -->|top-k = 2| E2{{expert 7}}:::model
+ R -. not chosen .-> E0[experts 1, 2, 4, 5, 6, 8<br/>idle for this token]:::ckpt
+ T --> SH{{shared expert<br/>every token}}:::model
+ E1 -->|weight p3| SUM((+)):::proc
+ E2 -->|weight p7| SUM
+ SH --> SUM
+ SUM --> Y([output]):::data
+ R -. how evenly were tokens spread? .-> AUX[load-balancing loss<br/>E x sum of f_e x P_e]:::loss
+ classDef data fill:#d6ffd9,stroke:#27ae60,stroke-width:2px,color:#143d1a;
+ classDef proc fill:#d6e8ff,stroke:#2c6fbb,stroke-width:2px,color:#0d2c52;
+ classDef model fill:#ffe8a3,stroke:#d48806,stroke-width:2px,color:#5a3d00;
+ classDef rl fill:#ffd9b3,stroke:#e67e22,stroke-width:2px,color:#6b3500;
+ classDef loss fill:#ffd6d6,stroke:#c0392b,stroke-width:2px,color:#5c1212;
+ classDef ckpt fill:#eeeeee,stroke:#555,stroke-width:2px,color:#222;