🎥 YouTube Videos

Transformers are the core neural network architecture behind modern AI systems, including Large Language Models (LLMs) such as GPT. Their primary objective is next-token prediction. Longer text generation happens by repeatedly predicting and appending the next token.

1.1 High-Level Data Flow in a Transformer

• 1. Tokenization and Embedding: Input text is split into tokens (words or subwords). Each token is converted into a high-dimensional embedding vector. These vectors are initially context-free and exist in spaces where directions can represent semantic concepts.
• 2. Iterative Processing (Transformer Layers): Each layer consists of two main blocks:
  • a) Attention Block: Allows tokens to exchange information and updates embeddings based on context.
  • b) MLP Block (Feed-Forward Layer): Processes each token independently, adding model capacity and potentially enabling storage of facts.
• 3. Prediction (Unembedding): After the final layer, the last token's vector is multiplied by the unembedding matrix to produce logits.
• 4. Softmax and Sampling: Softmax converts logits into probabilities. A temperature parameter controls randomness (Lower = deterministic, Higher = creative).

1.2 Parameter Distribution (GPT-3 Example)

• Total parameters: 175 billion
• Embedding + Unembedding matrices: ~1.2 billion
• Architecture is dominated by matrix–vector multiplications, enabling efficient training.

The attention mechanism is the key innovation that allows transformers to build context-aware representations.

2.1 Goal of Attention

• Transform embeddings from context-free to context-rich.
• Tokens dynamically influence one another.

2.2 Single Attention Head Mechanics

Each attention head is defined by three learned matrices: Query (Wᴽ), Key (Wᴷ), and Value (Wⱽ).

1. Query and Key Creation: Input embeddings are multiplied by Wᴽ (Queries - "What am I looking for?") and Wᴷ (Keys - "What do I contain?").
2. Attention Score Calculation: Compute dot products between all Q–K pairs to produce relevance scores.
3. Softmax Normalization: Applied column by column to produce attention weights (0 to 1).
4. Masking (Causal Models): Future tokens are masked to ensure later tokens cannot influence earlier ones.
5. Value Aggregation: Input embeddings are multiplied by Wⱽ. Attention weights are used to compute a weighted sum of Values, added to the original embedding.

2.3 Multi-Head Attention

Transformers use many attention heads in parallel. Example (GPT-3): 96 attention heads per layer, 96 layers total.

This video focuses on the MLP (Feed-Forward) blocks, which are believed to play a major role in storing factual knowledge.

3.1 Key Properties of MLP Blocks

• Operate independently on each token (No cross-token communication).
• Contain the majority of model parameters.

3.2 Structure of an MLP Block

1. Up-Projection (Wᵤₚ): Projects into a larger space (4x embedding size). Can be interpreted as probing for specific features.
2. Non-Linearity (ReLU): Sets negative values to 0, enabling combination of features.
3. Down-Projection (W_dₒwₙ): Maps back to embedding size. Columns define meaning change.
4. Residual Connection: Output is added back to original embedding (refining meaning).

3.4 Superposition Hypothesis

Models may store far more features than dimensions via vectors that are almost perpendicular. This explains why concepts are often distributed rather than localized.