Self-Attention

The Core Mechanism Behind Modern Language Models

What is Self-Attention?

Self-attention (also called intra-attention) is a mechanism that allows each position in a sequence to attend to all other positions within the same sequence. It's the fundamental building block of Transformer architectures and enables models to capture contextual relationships between tokens regardless of their distance in the sequence.

💡 Key Insight: Unlike RNNs that process sequences sequentially, self-attention computes relationships between all pairs of tokens in parallel, making it highly efficient for modern hardware and capable of capturing long-range dependencies.

How Self-Attention Works

For each token in the input sequence, self-attention creates three different representations through learned linear transformations:

Query

Represents "what am I looking for?" - what information this token needs from other tokens.

Key

Represents "what do I offer?" - what this token advertises about itself to others.

Value

Represents "what information do I actually provide?" - the actual content to be aggregated.

The Attention Formula

The self-attention mechanism computes:

Attention(Q, K, V) = softmax(QK^T / √d_k) · V

Step 1: Compute attention scores by taking dot product of Query with all Keys: QK^T

Step 2: Scale by √d_k (square root of key dimension) to prevent vanishing gradients

Step 3: Apply softmax to get attention weights (probabilities summing to 1)

Step 4: Multiply weights with Values to get weighted sum

Concrete Example

Let's see how self-attention works with the sentence: "The cat sat on the mat"

When processing "cat":

• Query from "cat" is compared with Keys from all words: ["The", "cat", "sat", "on", "the", "mat"]
• High attention scores might be given to: "sat" (action), "mat" (location)
• Lower attention to: "The", "on", "the" (less semantically relevant)
• The output is a weighted combination of Values, enriching "cat" with contextual information

Attention weights example (simplified):

The: 0.05 cat: 0.30 sat: 0.35 on: 0.08 the: 0.07 mat: 0.15

Key Properties

Parallelizable

All tokens are processed simultaneously, unlike sequential RNNs, enabling massive parallelization on GPUs.

Long-Range Dependencies

Can capture relationships between distant tokens directly, without the information bottleneck of RNNs.

Dynamic Weighting

Attention weights are computed dynamically based on content, not fixed like in traditional models.

Interpretable

Attention weights can be visualized to understand what the model is focusing on.

Computational Complexity

⚠️ Challenge: Standard self-attention has O(n²) time and memory complexity, where n is the sequence length. This quadratic scaling becomes a bottleneck for very long sequences.

This limitation has inspired numerous efficiency improvements:

Technique	Complexity	Description
Standard Attention	O(n²)	Full attention matrix computation
FlashAttention	O(n²)	IO-aware algorithm, faster in practice
Sparse Attention	O(n√n)	Attend to subset of tokens (Longformer, BigBird)
Linear Attention	O(n)	Approximate attention (Performer, Linformer)

Multi-Head Attention

In practice, Transformers use multi-head attention, which runs multiple self-attention operations in parallel. Each "head" can learn to focus on different types of relationships:

Head 1

Syntax

Head 2

Semantics

Head 3

Position

Head N

Coreference

head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)
MultiHead(Q,K,V) = Concat(head₁,...,head_h)W^O

Self-Attention Variants

🔒 Masked Self-Attention (Causal)

Used in decoder-only models (GPT). Each position can only attend to previous positions, preventing the model from "cheating" by looking at future tokens during training.

🔄 Cross-Attention

Queries come from one sequence, Keys and Values from another (e.g., attending from decoder to encoder output). Used in encoder-decoder models for tasks like translation.

🎯 Grouped-Query Attention (GQA)

Shares Key/Value heads across multiple Query heads to reduce memory usage during inference, particularly important for reducing KV-cache size in large models.

Why Self-Attention Matters

Self-attention is the foundation of modern AI breakthroughs:

Language Models

GPT, BERT, LLaMA, Claude - all powered by self-attention

Vision

Vision Transformers (ViT), DALL-E, Stable Diffusion

Audio & Speech

Whisper, speech recognition, music generation

Learn More

📚 Essential Resources

Test Your Knowledge

Score 8/10 or higher to pass