CI/CD Pipelines for GenAI

Automate testing, building, and deploying ML models and AI applications

What is CI/CD?

CI/CD (Continuous Integration / Continuous Deployment) is a set of practices that automate the process of integrating code changes, testing them, and deploying to production. For GenAI projects, CI/CD ensures that model updates, API changes, and infrastructure modifications are tested and deployed reliably and consistently.

Key Innovation: Modern CI/CD for ML includes model validation, performance benchmarking, and canary deployments—ensuring new model versions don't degrade production quality before full rollout.

Continuous Integration

Automatically test code on every commit. Catch bugs early before they reach production.

Continuous Delivery

Keep code in deployable state. Manual approval before production deployment.

Continuous Deployment

Automatic deployment to production after all tests pass. No manual intervention.

GenAI Pipeline Stages

1. Code

Trigger on push, PR, or schedule

2. Test

Unit, integration, linting

3. Build

Docker image, push to registry

4. Validate

Model accuracy, latency checks

5. Staging

Deploy to staging, E2E tests

6. Production

Canary or blue-green deploy

Complete GenAI Pipeline (GitHub Actions)

# .github/workflows/genai-cicd.yml
name: GenAI CI/CD Pipeline

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

env:
  PROJECT_ID: my-gcp-project
  SERVICE: genai-api
  REGION: us-central1

jobs:
  # Stage 1: Test
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
          cache: 'pip'
      
      - name: Install dependencies
        run: pip install -r requirements.txt -r requirements-dev.txt
      
      - name: Lint code
        run: |
          ruff check src/
          mypy src/
      
      - name: Run unit tests
        run: pytest tests/unit -v --cov=src --cov-report=xml
      
      - name: Run integration tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: pytest tests/integration -v

  # Stage 2: Build
  build:
    needs: test
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    outputs:
      image_tag: ${{ steps.build.outputs.image_tag }}
    
    steps:
      - uses: actions/checkout@v4
      
      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.GCP_SA_KEY }}
      
      - name: Set up Cloud SDK
        uses: google-github-actions/setup-gcloud@v2
      
      - name: Build and push image
        id: build
        run: |
          IMAGE_TAG="gcr.io/$PROJECT_ID/$SERVICE:${{ github.sha }}"
          docker build -t $IMAGE_TAG .
          docker push $IMAGE_TAG
          echo "image_tag=$IMAGE_TAG" >> $GITHUB_OUTPUT

  # Stage 3: Model Validation
  validate:
    needs: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Run model benchmarks
        env:
          MODEL_IMAGE: ${{ needs.build.outputs.image_tag }}
        run: |
          python scripts/benchmark_model.py \
            --image $MODEL_IMAGE \
            --min-accuracy 0.95 \
            --max-latency-p99 200ms

  # Stage 4: Deploy to Staging
  staging:
    needs: [build, validate]
    runs-on: ubuntu-latest
    environment: staging
    
    steps:
      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.GCP_SA_KEY }}
      
      - name: Deploy to Cloud Run (Staging)
        run: |
          gcloud run deploy $SERVICE-staging \
            --image ${{ needs.build.outputs.image_tag }} \
            --region $REGION \
            --no-traffic
      
      - name: Run E2E tests
        run: |
          STAGING_URL=$(gcloud run services describe $SERVICE-staging --region $REGION --format='value(status.url)')
          pytest tests/e2e --base-url=$STAGING_URL

  # Stage 5: Deploy to Production (Canary)
  production:
    needs: staging
    runs-on: ubuntu-latest
    environment: production
    
    steps:
      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.GCP_SA_KEY }}
      
      - name: Canary deployment (10%)
        run: |
          gcloud run deploy $SERVICE \
            --image ${{ needs.build.outputs.image_tag }} \
            --region $REGION \
            --tag canary \
            --no-traffic
          
          gcloud run services update-traffic $SERVICE \
            --region $REGION \
            --to-tags canary=10
      
      - name: Wait and monitor
        run: |
          sleep 300  # Wait 5 minutes
          python scripts/check_canary_health.py
      
      - name: Full rollout
        run: |
          gcloud run services update-traffic $SERVICE \
            --region $REGION \
            --to-latest

Testing Strategies for GenAI

🧪 Unit Tests

Test individual functions: prompt templates, data processing, utility functions

🔗 Integration Tests

Test API endpoints, database connections, external service calls

📊 Model Validation

Check accuracy, latency, and output quality against baseline

🌐 E2E Tests

Full user journey tests in staging environment

🔒 Security Scans

Container vulnerability scanning, dependency checks

⚡ Performance Tests

Load testing, latency benchmarks, resource usage

Deployment Strategies

🐤 Canary

Route 5-10% traffic to new version, monitor, then full rollout

Best for: High-risk changes, model updates

🔵🟢 Blue-Green

Two identical environments, instant traffic switch

Best for: Fast rollback needs

🔄 Rolling

Gradually replace old pods with new ones

Best for: Kubernetes deployments

Best Practices

Warning: Never deploy directly to production without staging validation. Model regressions can severely impact user experience and business metrics.

Environment protection: Require approvals for production deployments
Immutable artifacts: Tag images with commit SHA, never overwrite :latest in prod
Model versioning: Track model versions alongside code versions
Feature flags: Decouple deployment from feature release
Rollback plan: Always have one-click rollback capability
Monitoring: Set up alerts for error rates, latency, accuracy drift
Secrets rotation: Never hardcode secrets; use secret managers
Parallel jobs: Run independent tests in parallel for faster feedback

CI/CD Tools Comparison

Tool	Best For	Highlights
GitHub Actions	GitHub repositories	Native integration, marketplace
Cloud Build	GCP deployments	Deep GCP integration, serverless
GitLab CI	Self-hosted, all-in-one	Built into GitLab, DAG pipelines
Jenkins	Complex, custom pipelines	Highly customizable, plugin ecosystem
CircleCI	Fast builds, Docker	Performance, caching, insights