A/B Testing - Technical

What is A/B Testing for LLMs?

A/B testing (also called split testing) for LLMs is the practice of comparing two or more variants of prompts, models, or configurations to determine which performs better on specific metrics. It enables data-driven decisions about changes to your AI system.

Unlike traditional A/B testing, LLM experiments must account for output variability, subjective quality measures, and context-dependent performance.

What Can You A/B Test?

System Prompts

Test different instruction sets, personas, formatting rules, and constraints.

Model Versions

Compare GPT-4 vs Claude, or different versions like GPT-4 vs GPT-4-turbo.

Parameters

Temperature, top-p, max tokens, presence/frequency penalty settings.

RAG Configurations

Chunk sizes, retrieval strategies, number of context documents.

Few-Shot Examples

Number of examples, example selection, example ordering.

Chain Strategies

CoT vs direct, multi-step vs single-step, agent architectures.

Experiment Design

Define Hypothesis

Choose Metrics

Split Traffic

Run Experiment

Analyze Results

Key Principles

Single variable: Change only one thing at a time
Random assignment: Users/queries randomly assigned to variants
Sufficient sample size: Enough data for statistical significance
Pre-defined metrics: Decide what to measure before running

Common Pitfalls

Stopping tests too early ("peeking")
Testing multiple changes simultaneously
Ignoring variance in LLM outputs
Not accounting for user/query segments

Key Metrics for LLM A/B Tests

Category	Metric	Description	How to Measure
Quality	Relevance	Response addresses the query	LLM-as-Judge, Human eval
Quality	Accuracy	Factual correctness	Ground truth comparison
User	Thumbs Up Rate	Positive feedback ratio	In-app feedback buttons
User	Task Completion	User achieves their goal	Session analysis, surveys
Efficiency	Latency	Time to first/full response	Instrumentation, logs
Efficiency	Token Usage	Input/output tokens consumed	API response metadata
Safety	Harmful Output Rate	Toxic/unsafe responses	Content filters, moderation

Implementation Example

Traffic Splitting Router

import hashlib
import random
from dataclasses import dataclass
from typing import Dict, Any

@dataclass
class Variant:
    name: str
    weight: float  # 0.0 to 1.0
    config: Dict[str, Any]

class ABRouter:
    def __init__(self, experiment_name: str, variants: list[Variant]):
        self.experiment_name = experiment_name
        self.variants = variants
        
    def get_variant(self, user_id: str) -> Variant:
        """Deterministically assign user to variant based on user_id"""
        # Hash ensures same user always gets same variant
        hash_input = f"{self.experiment_name}:{user_id}"
        hash_value = int(hashlib.md5(hash_input.encode()).hexdigest(), 16)
        bucket = (hash_value % 100) / 100.0
        
        cumulative = 0.0
        for variant in self.variants:
            cumulative += variant.weight
            if bucket < cumulative:
                return variant
        return self.variants[-1]

# Example usage
experiment = ABRouter(
    experiment_name="prompt_v2_test",
    variants=[
        Variant("control", 0.5, {"prompt": "You are a helpful assistant."}),
        Variant("treatment", 0.5, {"prompt": "You are an expert AI assistant. Be concise."})
    ]
)

variant = experiment.get_variant(user_id="user_123")
print(f"User assigned to: {variant.name}")

Metrics Collection & Analysis

from scipy import stats
import pandas as pd

class ExperimentAnalyzer:
    def __init__(self, experiment_name: str):
        self.experiment_name = experiment_name
        self.results = []
    
    def log_result(self, variant: str, metrics: dict):
        """Log a single observation"""
        self.results.append({"variant": variant, **metrics})
    
    def analyze(self, metric: str, confidence: float = 0.95):
        """Compare variants using t-test"""
        df = pd.DataFrame(self.results)
        
        control = df[df["variant"] == "control"][metric]
        treatment = df[df["variant"] == "treatment"][metric]
        
        # Two-sample t-test
        t_stat, p_value = stats.ttest_ind(control, treatment)
        
        # Effect size (Cohen's d)
        pooled_std = ((control.std()**2 + treatment.std()**2) / 2) ** 0.5
        cohens_d = (treatment.mean() - control.mean()) / pooled_std
        
        return {
            "metric": metric,
            "control_mean": control.mean(),
            "treatment_mean": treatment.mean(),
            "lift": (treatment.mean() - control.mean()) / control.mean() * 100,
            "p_value": p_value,
            "significant": p_value < (1 - confidence),
            "effect_size": cohens_d,
            "sample_sizes": {"control": len(control), "treatment": len(treatment)}
        }

# Example
analyzer = ExperimentAnalyzer("prompt_v2_test")
result = analyzer.analyze("user_satisfaction", confidence=0.95)

if result["significant"]:
    print(f"✅ Significant! Treatment is {result['lift']:.1f}% better")
else:
    print(f"❌ Not significant (p={result['p_value']:.3f})")

Using PromptFoo for A/B Testing

# promptfooconfig.yaml
prompts:
  - id: control
    raw: "You are a helpful assistant. Answer the question: {{question}}"
  - id: treatment  
    raw: "You are an expert AI assistant. Be concise and accurate. Question: {{question}}"

providers:
  - openai:gpt-4

tests:
  - vars:
      question: "What is machine learning?"
    assert:
      - type: contains
        value: "algorithm"
      - type: llm-rubric
        value: "Response is clear and educational"
        
  - vars:
      question: "Explain quantum computing simply"
    assert:
      - type: llm-rubric
        value: "Response is understandable by a beginner"
      - type: cost
        threshold: 0.01

npx promptfoo eval to run the comparison

Sample Size Considerations

LLM experiments often require larger sample sizes due to output variability. Key factors:

Minimum Detectable Effect (MDE)

The smallest difference you want to detect. Smaller MDE = larger sample needed.

Statistical Power

Probability of detecting a real effect. Typically 80%. Higher power = more samples.

Baseline Variance

How variable your metric is. LLMs have high variance = need more samples.

Significance Level (α)

Acceptable false positive rate. Typically 5% (p < 0.05).

Rule of thumb: For LLM experiments, aim for at least 100-500 samples per variant for quality metrics, and 1000+ samples for subtle effects like user behavior changes.

Best Practices

User-Level Assignment

Assign at user level, not request level. Same user should always see the same variant.

Run Long Enough

Run for at least one full week to capture day-of-week effects and reach sample size.

Guardrail Metrics

Always monitor safety metrics even if not primary goal. Stop if guardrails are breached.

Document Everything

Record hypothesis, variants, metrics, dates, and decisions for future reference.

Plan for Rollback

Have a quick rollback mechanism if treatment causes unexpected issues.

Start Small

Start with 5-10% traffic to treatment, ramp up gradually if metrics look good.

Tools & Platforms

PromptFoo

CLI tool for prompt testing and comparison with built-in assertions.

LangSmith

Tracing and evaluation with experiments and A/B comparison.

Braintrust

End-to-end evaluation platform with experiment tracking.

Statsig

Feature flags and experimentation platform with statistical analysis.

LaunchDarkly

Feature management with experimentation and targeted rollouts.

PostHog

Open-source product analytics with A/B testing and feature flags.

A/B Testing for LLMs