Kubernetes for LLMs

Why Kubernetes for LLMs?

Auto-scaling

Scale replicas based on GPU utilization, queue depth, or custom metrics.

GPU Scheduling

NVIDIA device plugin, MIG support, multi-GPU pods, and GPU sharing.

Multi-cloud

Same config on GKE, EKS, AKS, or on-prem. Avoid vendor lock-in.

Self-healing

Automatic restarts, health checks, and rolling updates with zero downtime.

GPU Setup

1. Install NVIDIA Device Plugin

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.14.0/nvidia-device-plugin.yml

2. Verify GPUs Available

kubectl get nodes -o json | jq '.items[].status.capacity["nvidia.com/gpu"]'

vLLM Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: vllm-llama
  template:
    metadata:
      labels:
        app: vllm-llama
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - "--model"
          - "meta-llama/Llama-3.1-8B-Instruct"
          - "--port"
          - "8000"
        ports:
          - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            memory: "16Gi"
            cpu: "4"
        env:
          - name: HUGGING_FACE_HUB_TOKEN
            valueFrom:
              secretKeyRef:
                name: hf-token
                key: token
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-llama-svc
spec:
  selector:
    app: vllm-llama
  ports:
    - port: 8000
      targetPort: 8000
  type: ClusterIP

Ollama Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
          - containerPort: 11434
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            memory: "8Gi"
        volumeMounts:
          - name: ollama-data
            mountPath: /root/.ollama
      volumes:
        - name: ollama-data
          persistentVolumeClaim:
            claimName: ollama-pvc

Horizontal Pod Autoscaler

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: vllm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: vllm-llama
  minReplicas: 1
  maxReplicas: 4
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    # GPU metrics require DCGM exporter
    - type: Pods
      pods:
        metric:
          name: DCGM_FI_DEV_GPU_UTIL
        target:
          type: AverageValue
          averageValue: "80"

💡 Tip: Install DCGM Exporter + Prometheus Adapter for GPU-based autoscaling.

Ingress (NGINX)

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: llm-ingress
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "50m"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
spec:
  ingressClassName: nginx
  rules:
    - host: llm.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: vllm-llama-svc
                port:
                  number: 8000

Resource Recommendations

Model Size	GPU	Memory	Notes
7B (Q4)	1x T4 (16GB)	8Gi	Ollama, llama.cpp
7-13B (FP16)	1x A10G (24GB)	16Gi	vLLM recommended
30-40B	2x A10G or 1x A100	32Gi	Tensor parallelism
70B+	4x A100 (80GB)	64Gi	Pipeline parallelism

Useful Tools

KServe

Serverless ML inference on Kubernetes. Supports transformers, ONNX, and more.

Ray Serve

Distributed serving framework. Great for multi-model and batching.

Triton Inference Server

NVIDIA's high-performance inference server. TensorRT optimization.

GPU Operator

NVIDIA's K8s operator. Auto-installs drivers, device plugin, DCGM.