Vision Transformers (ViT)
Vision Transformers have revolutionized computer vision by treating images as sequences of patches, bringing the scalability and global context of Transformers to pixel data. They power state-of-the-art models like SAM and DINOv2.
The ViT Architecture
Patches as Tokens
Unlike CNNs that process pixels with sliding windows, ViT splits an image into fixed-size square patches (e.g., 16x16). Each patch is flattened and linearly projected into an embedding, becoming a "visual token" analogous to a word in NLP.
Global Self-Attention
Standard Transformer encoders process these tokens. The Self-Attention mechanism allows every patch to attend to every other patch instantly. This captures long-range dependencies and global context from the very first layer, something CNNs only achieve deep in the network.
Standard ViT Pipeline
ViT vs CNN (Inductive Bias)
| Feature | CNNs (ResNet, EfficientNet) | Vision Transformers (ViT) |
|---|---|---|
| Inductive Bias | High (Translation Equivariance, Locality) | Low (Learned from data) |
| Data Efficiency | Good on small datasets | Data hungry (Needs ImageNet-21k+) |
| Global Context | Gradual (Receptive field grows with depth) | Immediate (All-to-all attention) |
| Scaling | Saturates eventually | Scales impressively (ViT-22B) |
Hybrid Models: Because ViTs lack inductive bias, they are harder to train on small data. Many modern architectures (like Swin Transformer or ConvNeXt) mix Convolutional inductive biases with Transformer Self-Attention to get the best of both worlds.
State-of-the-art (2024-2025)
DINOv2 (Meta AI) Self-Supervised
A foundational model trained on 142M images without labels. DINOv2 learns such powerful features that it can perform semantic segmentation, depth estimation, and retrieval out-of-the-box using just a simple linear classifier (Frozen Features). It solves the "data labeling bottleneck."
Segment Anything Model (SAM) Zero-Shot
Uses a heavy ViT Image Encoder and a lightweight Prompt Encoder. SAM can "cut out" any object in any image based on a click, box, or text prompt. It demonstrates true zero-shot transfer to new domains (e.g., medical imaging, satellite) without retraining.
ViT-22B (Google) Scale
Proves that vision models scale like LLMs. With 22 billion parameters, it achieves state-of-the-art performance and begins to show human-like shape/texture bias, moving away from the "texture bias" flaw of standard CNNs.