Skip to content

Build a modern LLM from scratch. Every line commented. Explained like we are five.

12.2 KB
Raw Blame

Chapter 4 — Positional Encoding: Teaching Order

The 5-Year-Old Analogy

Consider two sentences:

  • "The dog bit the man." — scary
  • "The man bit the dog." — weird

Same words, different order -> completely different meaning.

But the Transformer reads all words at once (not one by one like humans do). It has no idea which word comes first! So we must stamp each word with its position before feeding it to the model.

The Three Generations of Position Encoding

Method How It Works Pros Cons Used By
Learned Each position gets its own learned vector Simple, flexible Can't handle sequences longer than training GPT-2, BERT
Sinusoidal Fixed sine/cosine waves by position Works for any length Weaker at relative positions Original Transformer
RoPE Rotates Q,K vectors by position angle Perfect relative positions, any length Slightly more complex LLaMA, Mistral, Qwen, Gemma
ALiBi Adds a bias to attention scores based on distance No learned params, very fast Less expressive BLOOM, MPT

Modern Approach: Rotary Position Embeddings (RoPE)

A Quick Word on "Query" and "Key" Vectors

RoPE acts on two vectors you haven't met yet: the query and the key. They belong to attention, which is the whole of Chapter 5. Here is all you need for now:

  • Inside attention, every token's embedding gets turned into a query vector ("what am I looking for?") and a key vector ("what do I have to offer?").
  • Attention compares a query with a key by taking their dot product. A big result means "pay attention to this token".
  • RoPE rotates the query and key vectors right before that dot product, so the result also depends on how far apart the two tokens are.

So for this chapter, treat Q and K as "two vectors that get compared with a dot product". Chapter 5 builds them step by step.

The Idea

Instead of adding position numbers to embeddings, RoPE rotates the query and key vectors by an angle that depends on position.

The Math Intuition

In 2D, rotating a vector (x, y) by angle θ gives:

x' = x*cos(θ) - y*sin(θ)
y' = x*sin(θ) + y*cos(θ)

RoPE splits each query and key vector into adjacent, non-overlapping pairs of dimensions: (0,1), (2,3), (4,5), ... and rotates each pair as if it were its own little 2D vector. A vector with d_model=4 has exactly two pairs: (0,1) and (2,3). It is NOT every possible combination of two dimensions. The rotation angle for position p and pair i (dimensions 2i, 2i+1) is:

θ(p, i) = p / (10000^(2i/d_model))

Key insight: The angle depends on p (position) and i (dimension pair index). Lower dimension pairs rotate FAST (capturing local word relationships). Higher pairs rotate SLOW (capturing long-range relationships).

Numerical Worked Example

Let's trace RoPE with a tiny model: d_model=4, processing position p=1:

Step 1: Compute frequencies for each dimension pair

Pair 0 (dims 0,1): freq = 1 / 10000^(0/4)   = 1 / 1       = 1.000
Pair 1 (dims 2,3): freq = 1 / 10000^(2/4)   = 1 / 10000^0.5 = 1 / 100 = 0.010

Step 2: Compute rotation angle for position p=1

Pair 0 angle: θ₀ = p * freq₀ = 1 * 1.000 = 1.000 radian (≈ 57.3°)
Pair 1 angle: θ₁ = p * freq₁ = 1 * 0.010 = 0.010 radian (≈ 0.57°)

Step 3: Apply rotation to a query vector at position 1

Before RoPE: q₁ = [0.8, 0.3, -0.5, 0.2]

Rotate pair 0 (dims 0,1) by 57.3°:
  dim0' = 0.8*cos(1.0) - 0.3*sin(1.0) = 0.8*0.540 - 0.3*0.842 = 0.432 - 0.253 = 0.179
  dim1' = 0.8*sin(1.0) + 0.3*cos(1.0) = 0.8*0.842 + 0.3*0.540 = 0.674 + 0.162 = 0.836

Rotate pair 1 (dims 2,3) by 0.57°:
  dim2' = -0.5*cos(0.01) - 0.2*sin(0.01) = -0.5*1.000 - 0.2*0.010 = -0.500 - 0.002 = -0.502
  dim3' = -0.5*sin(0.01) + 0.2*cos(0.01) = -0.5*0.010 + 0.2*1.000 = -0.005 + 0.200 = 0.195

After RoPE: q₁' = [0.179, 0.836, -0.502, 0.195]

Now let's compute what happens at positions 1 and 3:

Position 1: θ₀ = 1.0 rad,  θ₁ = 0.01 rad
Position 3: θ₀ = 3.0 rad,  θ₁ = 0.03 rad

The dot product q₁ · k₃ will depend on the DIFFERENCE:
  Δθ₀ = 3.0 - 1.0 = 2.0 rad
  Δθ₁ = 0.03 - 0.01 = 0.02 rad
  
This difference depends ONLY on (3-1)=2, the relative distance!
Absolute positions don't matter — only how far apart they are.

This is why RoPE is brilliant: the attention score between position i and j depends only on their relative distance (j-i), not their absolute positions.

Why theta=10000?

The base frequency theta = 10000 controls the "spread" of frequencies:

Low theta (e.g., 100):
  - All dimension pairs rotate similarly
  - Model is more "position-agnostic" — better for long contexts
  - But loses fine-grained position resolution

High theta (e.g., 100000):
  - Very different rotation speeds across dimensions
  - Better at distinguishing nearby positions
  - But struggles with very long contexts

10000 was found empirically to balance these tradeoffs.

Extending Context Beyond Training Length

What if we trained on 2048 tokens but want to use 4096 at inference?

The problem: RoPE was precomputed for positions 0-2047. Position 3000 was never seen.

Solutions:

Method How It Works Quality
Linear interpolation Position / scale (e.g., p/2 for 2x length) OK, loses resolution
NTK-aware scaling Scale theta differently per frequency Good
YaRN NTK + temperature scaling Best (used in production)
Retrain Just train on longer sequences Perfect but expensive

For our small training run, this doesn't matter — but know that it's a hot research area for production models.

RoPE Code — Annotated

import torch
import torch.nn as nn
import math


class RotaryPositionalEmbedding(nn.Module):
    """
    WHAT: Rotary Position Embeddings (RoPE).
    WHY: Instead of ADDING position info to embeddings,
         we ROTATE Q and K vectors by position-dependent angles.
         The dot product q_i · k_j then depends ONLY on (j-i),
         which is exactly what attention should care about.

         Paper: "RoFormer" (Su et al., 2021)
         Used in: LLaMA 1/2/3, Mistral, Mixtral, Qwen 1/2, Gemma

         How it works at a glance:
         1. For each pair of dimensions (0,1), (2,3), (4,5), ...
         2. Rotate by angle = position * frequency
         3. Lower dims rotate fast (local position)
            Higher dims rotate slow (global position)
         4. The dot product naturally depends on relative distance
    """

    def __init__(self, d_model: int, max_seq_len: int = 2048, theta: float = 10000.0):
        """
        WHAT: Precompute rotation frequencies for fast lookup.

        Args:
            d_model:     Head dimension (e.g., 64 for GPT-2). Must be even.
            max_seq_len: Precompute angles for positions 0..max_seq_len-1.
            theta:       Base frequency. 10000 is standard. Controls the
                         spread between fast and slow rotation frequencies.
        """
        super().__init__()

        # WHAT: Verify d_model is even (must have pairs to rotate)
        assert d_model % 2 == 0, (
            f"d_model ({d_model}) must be even for RoPE. "
            f"Each pair of dimensions needs a partner to rotate with."
        )

        # WHAT: Create dimension indices: [0, 2, 4, ..., d_model-2]
        # WHY: Each pair (2i, 2i+1) gets the same rotation frequency.
        #      We only need half the indices because pairs share.
        dim_indices = torch.arange(0, d_model, 2).float()

        # WHAT: Compute rotation frequencies
        # WHY: theta_i = 1 / (theta ^ (2i / d_model))
        #
        #      i=0:  1 / 10000^(0/64)      = 1.0      → fast rotation (local)
        #      i=30: 1 / 10000^(60/64)     ≈ 0.0001   → slow rotation (global)
        #
        #      This multi-scale approach means some dimensions
        #      capture local word order while others capture
        #      long-range position relationships.
        inv_freq = 1.0 / (theta ** (dim_indices / d_model))

        # WHAT: Precompute angles for all positions
        # WHY: Computing cos/sin during training is expensive.
        #      Precomputing them once and caching is 100x faster.
        positions = torch.arange(max_seq_len).float()     # [0, 1, 2, ..., 2047]

        # WHAT: Outer product: each position x each frequency
        #       freqs[p, i] = p * inv_freq[i] = angle for position p, dim pair i
        #       Shape: [max_seq_len, d_model/2]
        freqs = torch.outer(positions, inv_freq)

        # WHAT: Duplicate to full dimension
        # WHY: Each dim pair (2i, 2i+1) gets the same angle,
        #      so we copy each angle: [θ0, θ1, θ2, ...] -> [θ0, θ0, θ1, θ1, ...]
        emb = freqs.repeat_interleave(2, dim=-1)         # [max_seq_len, d_model]

        # WHAT: Cache cos and sin for all positions
        # WHY: register_buffer means these move with model.to(device)
        #      and are saved with model.state_dict(), but are NOT
        #      trainable parameters (no gradients needed).
        self.register_buffer("cos_cached", emb.cos())   # cos of each angle
        self.register_buffer("sin_cached", emb.sin())   # sin of each angle

    @staticmethod
    def rotate_half(x: torch.Tensor) -> torch.Tensor:
        """
        WHAT: Prepare a vector for the rotation formula.
        WHY:  The rotation formula is: x' = x*cos + rotate_half(x)*sin

              For vector [x0, x1, x2, x3, x4, x5]:
              rotate_half returns [-x1, x0, -x3, x2, -x5, x4]

              Why this works: For pair (x0, x1) rotated by angle θ:
                x0' = x0*cos(θ) - x1*sin(θ)   ← matches: x0*cos + (-x1)*sin
                x1' = x0*sin(θ) + x1*cos(θ)   ← matches: x1*cos + (x0)*sin

              So executing (x*cos + rotate_half(x)*sin) rotates every
              adjacent pair at once — no loop needed!
        """
        x_even = x[..., 0::2]   # dims 0, 2, 4, ... — the "x" of each pair
        x_odd  = x[..., 1::2]   # dims 1, 3, 5, ... — the "y" of each pair

        # WHAT: Interleave back into [-x1, x0, -x3, x2, ...]
        # WHY: The cos/sin caches are stored as [θ0, θ0, θ1, θ1, ...],
        #      so each pair must stay next to its own angle.
        return torch.stack([-x_odd, x_even], dim=-1).flatten(-2)

    def forward(self, x: torch.Tensor, seq_len: int) -> torch.Tensor:
        """
        WHAT: Apply RoPE to queries or keys.

        Input:  [batch, num_heads, seq_len, head_dim]
                x can be either Q or K (NOT V — values don't need position)
        Output: Same shape, rotated by position-dependent angles

        WHY applied only to Q and K:
        The attention score = Q_i · K_j controls WHICH values to attend to.
        We want this score to depend on relative position.
        The VALUE vectors carry content — position is irrelevant for the
        content itself. Position only matters for deciding which tokens
        to pay attention TO.
        """
        # WHAT: Extract cos and sin for current sequence length
        # WHY: If seq_len=512 but max_seq_len=2048, we only need
        #      the first 512 rows of the cached cos/sin tables.
        cos = self.cos_cached[:seq_len]   # [seq_len, head_dim]
        sin = self.sin_cached[:seq_len]   # [seq_len, head_dim]

        # WHAT: Add batch and head dimensions for broadcasting
        # WHY: cos/sin are [seq_len, head_dim]. We need them to
        #      multiply with x [batch, heads, seq_len, head_dim].
        #      unsqueeze(0).unsqueeze(0) adds dims at positions 0 and 1:
        #      [seq_len, head_dim] -> [1, 1, seq_len, head_dim]
        #      Now they broadcast correctly over batch and heads.
        cos = cos.unsqueeze(0).unsqueeze(0)
        sin = sin.unsqueeze(0).unsqueeze(0)

        # WHAT: Execute rotation: x_rotated = x*cos(θ) + rotate_half(x)*sin(θ)
        # WHY: This is mathematically equivalent to applying a 2D rotation
        #      matrix to each pair of dimensions, but implemented in pure
        #      element-wise operations — much faster and parallelizable.
        return (x * cos) + (self.rotate_half(x) * sin)

Previous: Chapter 3 — Embeddings Next: Chapter 5 — Attention