Model Distillation

What is Knowledge Distillation?

Knowledge Distillation is a technique where a smaller "student" model learns to mimic the behavior of a larger, more powerful "teacher" model. Instead of training from scratch on raw data, the student learns from the teacher's outputs, capturing its reasoning patterns and knowledge.

This is one of the most practical ways to deploy LLMs in production: use GPT-4 or Claude to generate training data, then distill that knowledge into a Llama 8B or Mistral 7B for cost-effective inference.

How Distillation Works

1. Generate Data

Use the teacher model to generate high-quality responses for your target task.

2. Fine-tune Student

Train a smaller model on this data using supervised fine-tuning (SFT).

3. Deploy Student

Use the distilled model in production at a fraction of the cost.

Types of Distillation

Response Distillation (Most Common)

The student learns to produce the same text outputs as the teacher. Simple and effective.

# Generate training data from teacher
training_data = []
for prompt in prompts:
    response = gpt4.generate(prompt)
    training_data.append({"prompt": prompt, "response": response})

# Fine-tune student on this data
student = finetune(llama_8b, training_data)

Soft Label Distillation

Student learns the teacher's probability distribution over tokens (logits), not just the final output. Captures more nuanced knowledge.

# Loss = α * CE(student_logits, hard_labels) + 
#        (1-α) * KL(student_logits/T, teacher_logits/T)
# T = temperature (higher = softer distribution)
# α = weight between hard and soft targets

Chain-of-Thought Distillation

Generate reasoning traces with the teacher, then train the student on both the reasoning and the answer. Improves student's reasoning abilities.

Practical Example: Distilling GPT-4

from openai import OpenAI
from datasets import Dataset
import json

# Step 1: Generate training data from GPT-4
client = OpenAI()
training_examples = []

prompts = load_your_task_prompts()  # Your domain-specific prompts

for prompt in prompts:
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": prompt}
        ],
        temperature=0.7
    )
    
    training_examples.append({
        "prompt": prompt,
        "response": response.choices[0].message.content
    })

# Step 2: Format for fine-tuning
def format_for_chat(example):
    return {
        "messages": [
            {"role": "user", "content": example["prompt"]},
            {"role": "assistant", "content": example["response"]}
        ]
    }

dataset = Dataset.from_list(training_examples)
dataset = dataset.map(format_for_chat)

# Step 3: Fine-tune with your framework of choice
# Options: Axolotl, Unsloth, TRL, OpenAI fine-tuning
dataset.save_to_disk("distillation_data")

💡 Pro Tip: Use 5,000-50,000 examples for most tasks. More examples = better student performance, but with diminishing returns after ~50k.

Famous Distilled Models

Model	Teacher	Student Size	Notable Achievement
Alpaca	text-davinci-003	Llama 7B	First viral open instruction model
Vicuna	ChatGPT	Llama 13B	90% of ChatGPT quality
Orca	GPT-4	Llama 13B	Explanation-based distillation
Phi-2/3	GPT-4 (synthetic)	2.7B / 3.8B	Textbook-quality synthetic data
Zephyr	GPT-4 + UltraFeedback	Mistral 7B	DPO-aligned distillation

Best Practices

Diverse prompts: Include edge cases and variations in your training data
High-quality teacher: Use the best available model; garbage in = garbage out
Temperature variation: Generate multiple responses per prompt at different temperatures
Filter bad outputs: Remove low-quality or incorrect teacher responses before training
Evaluate rigorously: Test student on held-out set; compare to teacher performance

⚠️ Legal Note: Check the teacher model's Terms of Service. Some providers (like OpenAI) have restrictions on using outputs to train competing models.