Batching Strategies
Maximize GPU utilization and throughput by processing multiple requests together. Learn static, dynamic, and continuous batching techniques for production LLM serving.
🎯 Why Batching Matters
❌ Without Batching
- • Process 1 request at a time
- • GPU utilization: 10-30%
- • High cost per token
- • Wasted compute resources
✅ With Batching
- • Process 8-256 requests together
- • GPU utilization: 80-95%
- • 5-20x higher throughput
- • Significantly lower cost per token
💡 Key Insight: LLM inference is memory-bandwidth bound. Matrix multiplications can process multiple sequences simultaneously with minimal overhead, making batching extremely efficient.
📊 Batching Strategies Comparison
| Strategy | How It Works | Throughput | Latency | Use Case |
|---|---|---|---|---|
| Static Batching | Wait for N requests, process together | Medium | High (waiting) | Batch processing jobs |
| Dynamic Batching | Timeout-based collection | Good | Medium | API endpoints |
| Continuous Batching | Add/remove requests mid-generation | Excellent | Low | Production serving |
| Iteration-level Batching | Continuous + smart scheduling | Best | Lowest | High-traffic APIs |
🔄 Continuous Batching (In-flight Batching)
The gold standard for production LLM serving. Unlike static batching, requests can enter and leave the batch at any token generation step.
How It Works
- 1 New requests immediately join the batch
- 2 Each iteration generates 1 token per sequence
- 3 Completed sequences exit immediately
- 4 GPU slot freed for next waiting request
Benefits
- No waiting for batch to fill
- Short responses don't wait for long ones
- Near-optimal GPU utilization
- Built into vLLM, TensorRT-LLM, SGLang
Time → T1 T2 T3 T4 T5 T6 T7
Req A: [████][████][████][████][DONE]
Req B: [████][████][████][████][████][DONE]
Req C: [████][████][DONE]
Req D: [████][████][████]...
# Each [████] = one token generated
# Requests enter/exit dynamically📄 PagedAttention (vLLM)
The breakthrough that makes continuous batching memory-efficient. Manages KV cache like virtual memory pages.
How It Works
- • KV cache split into fixed-size "pages" (blocks)
- • Pages allocated on-demand, not pre-allocated
- • Sequences can share pages (prefix caching)
- • Memory fragmentation virtually eliminated
💻 Implementation Examples
vLLM Continuous Batching (Default)
# vLLM uses continuous batching by default
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Llama-3.1-8B-Instruct",
max_num_seqs=256, # Max concurrent sequences
max_num_batched_tokens=8192 # Tokens per iteration
)
# vLLM automatically batches these requests
prompts = [f"Question {i}: ..." for i in range(100)]
outputs = llm.generate(prompts, SamplingParams(max_tokens=100))vLLM Server API with Auto-Batching
# Start server with batching config
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--max-num-seqs 256 \
--max-num-batched-tokens 8192
# Send concurrent requests - they auto-batch
import asyncio
import aiohttp
async def send_request(session, prompt):
async with session.post(
"http://localhost:8000/v1/completions",
json={"model": "...", "prompt": prompt, "max_tokens": 100}
) as resp:
return await resp.json()⚙️ Key Batching Parameters
| Parameter | Description | Typical Range | Impact |
|---|---|---|---|
| max_num_seqs | Max concurrent sequences | 64-512 | ↑ throughput, ↑ memory |
| max_num_batched_tokens | Tokens processed per step | 2048-16384 | ↑ throughput, ↑ latency |
| gpu_memory_utilization | Target GPU memory usage | 0.8-0.95 | ↑ = more concurrent requests |
| block_size | KV cache page size | 16-32 | Lower = less memory waste |
✅ Best Practices
For Throughput
- Use continuous batching (vLLM, TensorRT-LLM)
- Increase max_num_seqs until memory is 90%+ used
- Enable prefix caching for repeated prompts
- Use quantization to fit more sequences
For Latency
- Lower max_num_batched_tokens for faster TTFT
- Enable streaming for perceived speed
- Use speculative decoding for faster generation
- Consider dedicated instance for low-latency tier