GenAIHub
EN | PT
← Back to Technical Section

Kubernetes for GenAI Workloads

Orchestrate and scale containerized ML models in production

What is Kubernetes?

Kubernetes (K8s) is an open-source container orchestration platform that automates deployment, scaling, and management of containerized applications. For GenAI workloads, Kubernetes provides the infrastructure to run inference endpoints at scale, manage GPU resources, and ensure high availability for ML services.

Key Innovation: Kubernetes enables horizontal pod autoscaling, self-healing deployments, and rolling updates—critical for production ML systems that need to handle variable inference loads while maintaining uptime.

Auto-scaling

HPA & VPA

Self-healing

Auto-restart pods

GPU Support

NVIDIA device plugin

Rolling Updates

Zero-downtime deploys

Core Concepts

Pod

Smallest deployable unit, runs containers

Deployment

Manages replica sets and updates

Service

Exposes pods via stable endpoint

Ingress

HTTP routing to services

Ingress Service Deployment Pod 1 Pod 2 Container Container HPA

GenAI Deployment Manifests

Deployment for Inference API

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: genai-inference
  labels:
    app: genai-inference
spec:
  replicas: 3
  selector:
    matchLabels:
      app: genai-inference
  template:
    metadata:
      labels:
        app: genai-inference
    spec:
      containers:
      - name: inference
        image: gcr.io/my-project/genai-api:v1.2.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
        env:
        - name: OPENAI_API_KEY
          valueFrom:
            secretKeyRef:
              name: api-secrets
              key: openai-key
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

Service & Ingress

# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: genai-inference-svc
spec:
  selector:
    app: genai-inference
  ports:
  - port: 80
    targetPort: 8080
  type: ClusterIP
---
# ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: genai-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
spec:
  tls:
  - hosts:
    - api.mygenai.com
    secretName: genai-tls
  rules:
  - host: api.mygenai.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: genai-inference-svc
            port:
              number: 80

GPU Workloads for ML Training

Tip: Use node selectors or taints/tolerations to schedule GPU workloads on nodes with NVIDIA GPUs. Install NVIDIA device plugin for GPU resource management. 

# gpu-training-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: model-training
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: trainer
        image: gcr.io/my-project/model-trainer:latest
        resources:
          limits:
            nvidia.com/gpu: 2  # Request 2 GPUs
          requests:
            memory: "16Gi"
            cpu: "4000m"
        volumeMounts:
        - name: training-data
          mountPath: /data
        - name: model-output
          mountPath: /models
      nodeSelector:
        cloud.google.com/gke-accelerator: nvidia-tesla-t4
      tolerations:
      - key: nvidia.com/gpu
        operator: Exists
        effect: NoSchedule
      volumes:
      - name: training-data
        persistentVolumeClaim:
          claimName: training-data-pvc
      - name: model-output
        persistentVolumeClaim:
          claimName: model-output-pvc

Horizontal Pod Autoscaler (HPA)

# hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: genai-inference-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: genai-inference
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300  # Wait 5 min before scaling down
    scaleUp:
      stabilizationWindowSeconds: 60   # Scale up faster

Essential kubectl Commands

# Apply manifests
kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

# View resources
kubectl get pods
kubectl get deployments
kubectl get services

# Describe resource details
kubectl describe pod genai-inference-xxx

# View logs
kubectl logs -f genai-inference-xxx

# Execute command in pod
kubectl exec -it genai-inference-xxx -- /bin/bash

# Scale deployment
kubectl scale deployment genai-inference --replicas=5

# Rolling update
kubectl set image deployment/genai-inference inference=gcr.io/my-project/genai-api:v1.3.0

# Rollback
kubectl rollout undo deployment/genai-inference

# View HPA status
kubectl get hpa

Best Practices for GenAI

Warning: Always set resource requests and limits for ML workloads. Without limits, a single inference request could consume all node resources.

  • Resource limits: Set memory/CPU requests and limits to prevent resource starvation
  • Health probes: Configure liveness and readiness probes for self-healing
  • Secrets management: Use Kubernetes Secrets or external secret stores (Vault)
  • Namespaces: Separate environments (dev, staging, prod) using namespaces
  • Pod Disruption Budgets: Ensure minimum replicas during updates
  • Node affinity: Schedule GPU workloads on appropriate node pools
  • ConfigMaps: Externalize configuration from container images
  • Monitoring: Use Prometheus + Grafana for metrics and alerting

Managed Kubernetes Options

GKE (Google)

Google Kubernetes Engine with Autopilot mode, tight GCP integration

EKS (AWS)

Elastic Kubernetes Service with Fargate serverless option

AKS (Azure)

Azure Kubernetes Service with Azure integration

Learn More

Essential Resources

Related Topics

Test Your Knowledge

Score 8/10 or higher to pass