Docker for LLMs

What is Docker and Why Use It for LLMs?

Docker is a containerization platform that packages applications and all their dependencies into standardized units called containers. Think of a container as a lightweight, portable "box" that contains everything your LLM needs to run: the model code, Python libraries, CUDA drivers, and system configurations.

For LLM deployments, this solves a critical challenge: LLMs have complex dependencies. A typical LLM inference setup requires specific versions of PyTorch, HuggingFace Transformers, CUDA, cuDNN, and dozens of other libraries. Without Docker, getting these working across different machines often leads to frustrating "dependency hell" and the infamous "it works on my machine" problem.

With Docker, you define your environment once, and it runs identically everywhere—on your laptop, your colleague's workstation, staging servers, or production cloud infrastructure.

Key Advantages

When Should You Use Docker for LLMs?

Setting Up GPU Access in Docker

By default, Docker containers cannot access the host's GPU. To run LLMs with GPU acceleration, you need to install the NVIDIA Container Toolkit. This allows Docker to pass through GPU resources to containers, enabling CUDA-based inference.

# Add NVIDIA package repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install the toolkit
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit

# Configure Docker to recognize NVIDIA runtime
sudo nvidia-ctk runtime configure --runtime=docker

# Restart Docker to apply changes
sudo systemctl restart docker

# This should display your GPU info inside the container
docker run --rm --gpus all nvidia/cuda:12.0-base nvidia-smi

Running Ollama in Docker

Ollama is the easiest way to run LLMs locally. The Docker image provides a ready-to-use server with GPU support. This is ideal for development, testing, or small-scale deployments.

# Start Ollama server with GPU support
docker run -d --gpus all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  --name ollama \
  ollama/ollama

# Download and run a model (inside the container)
docker exec -it ollama ollama run llama3.1

# Or use the API directly
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1",
  "prompt": "Explain Docker in simple terms"
}'

Running vLLM in Docker (Production)

vLLM is the industry-standard for high-throughput LLM inference. It uses PagedAttention to maximize GPU utilization and provides an OpenAI-compatible API. This is the recommended choice for production deployments.

# Run vLLM with OpenAI-compatible API
docker run -d --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
  --name vllm \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --port 8000

# Test with curl (OpenAI-compatible API)
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Text Generation Inference (TGI)

TGI is HuggingFace's production inference server. It's optimized for HuggingFace models and includes features like quantization, flash attention, and tensor parallelism. Great if you're already in the HuggingFace ecosystem.

# Run TGI with a HuggingFace model
docker run -d --gpus all \
  -v ~/.cache/huggingface:/data \
  -p 8080:80 \
  -e MODEL_ID=meta-llama/Llama-3.1-8B-Instruct \
  -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \
  --name tgi \
  ghcr.io/huggingface/text-generation-inference:latest

# With AWQ quantization (4-bit, uses less VRAM)
docker run -d --gpus all \
  -v ~/.cache/huggingface:/data \
  -p 8080:80 \
  -e MODEL_ID=TheBloke/Llama-2-7B-AWQ \
  -e QUANTIZE=awq \
  ghcr.io/huggingface/text-generation-inference:latest

Docker Compose for Production

Docker Compose defines your entire stack in a single file. It's easier to manage than long docker run commands and enables features like health checks, automatic restarts, and resource limits.

# docker-compose.yml
version: "3.8"

services:
  vllm:
    image: vllm/vllm-openai:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    volumes:
      - huggingface-cache:/root/.cache/huggingface
    ports:
      - "8000:8000"
    environment:
      - HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}
    command: >
      --model meta-llama/Llama-3.1-8B-Instruct
      --port 8000
      --gpu-memory-utilization 0.9
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

volumes:
  huggingface-cache:

docker compose up -d

Multi-GPU Configuration

For large models (30B+), you may need multiple GPUs. Docker supports selecting specific GPUs or using all available GPUs for tensor parallelism.

docker run --gpus all ...

# Use only GPU 0 and GPU 1
docker run --gpus '"device=0,1"' ...

docker run --gpus all vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4

Best Practices

Related Topics