LLM Sampling Parameters

Understanding Token Generation

Large Language Models generate text one token at a time. At each step, the model outputs a probability distribution over all possible next tokens (typically 32,000–100,000+ tokens in the vocabulary). Sampling parameters control how the next token is selected from this distribution, directly affecting the creativity, coherence, and predictability of outputs.

Understanding these parameters is essential for fine-tuning model behavior for different use cases—from deterministic code generation to creative writing.

The Token Generation Process

1. Forward Pass: Input tokens are processed through the transformer layers
2. Logits: The model outputs raw scores (logits) for each vocabulary token
3. Softmax: Logits are converted to a probability distribution (sums to 1.0)
4. Sampling: Parameters modify this distribution and select the next token
5. Repeat: The selected token is appended, and the process repeats

Temperature

Temperature scales the logits before softmax, controlling the "sharpness" of the probability distribution. It's the most commonly adjusted parameter.

# Temperature scaling formula
P(token_i) = exp(logit_i / T) / Σ exp(logit_j / T)

Where T = temperature
- T < 1.0: Sharpens distribution (more deterministic)
- T = 1.0: No change (default)
- T > 1.0: Flattens distribution (more random)
- T → 0:   Approaches greedy decoding (always pick highest probability)
            

Top-p (Nucleus Sampling)

Top-p dynamically selects the smallest set of tokens whose cumulative probability exceeds the threshold p. Only tokens in this "nucleus" are considered for sampling.

# Top-p algorithm
1. Sort tokens by probability (descending)
2. Compute cumulative probability
3. Keep tokens until cumulative prob ≥ p
4. Renormalize remaining probabilities
5. Sample from this reduced set

Example (top_p = 0.9):
Token     Prob    Cumulative
"the"     0.40    0.40        ✓
"a"       0.25    0.65        ✓
"an"      0.15    0.80        ✓
"this"    0.10    0.90        ✓ (threshold)
"that"    0.05    0.95        ✗ (excluded)
"one"     0.03    0.98        ✗
...
            

Top-k Sampling

Top-k simply keeps only the k highest-probability tokens and samples from them. It's simpler than top-p but less adaptive.

# Top-k algorithm
1. Sort tokens by probability (descending)
2. Keep only the top k tokens
3. Renormalize their probabilities
4. Sample from this set

Example (top_k = 3):
Token     Prob    
"the"     0.40    ✓ (kept)
"a"       0.25    ✓ (kept)
"an"      0.15    ✓ (kept)
"this"    0.10    ✗ (excluded)
"that"    0.05    ✗
...
            

Other Important Parameters

# Common settings
max_tokens=256   # Short responses (summaries, answers)
max_tokens=1024  # Medium responses (explanations)
max_tokens=4096  # Long responses (articles, code)
                    

stop=["\n\n", "###", "", "Human:"]
# Stops when any of these appear in output
                    

# Adjusted logit = logit - frequency_penalty * token_count
frequency_penalty=0.0  # No penalty (default)
frequency_penalty=0.5  # Mild reduction of common tokens
frequency_penalty=1.0  # Strong anti-repetition
                    

# Adjusted logit = logit - presence_penalty * (1 if token appeared else 0)
presence_penalty=0.0  # No penalty (default)
presence_penalty=0.6  # Encourage topic diversity
                    

seed=42  # Same seed + same input = same output (mostly)
# Note: Not all providers guarantee perfect reproducibility
                    

Recommended Parameter Combinations

API Examples

OpenAI

from openai import OpenAI
client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a haiku about Python"}],
    temperature=0.8,
    top_p=0.95,
    max_tokens=100,
    frequency_penalty=0.3,
    presence_penalty=0.0,
    stop=["\n\n"],
    seed=42
)
            

Anthropic (Claude)

import anthropic
client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Write a haiku about Python"}],
    temperature=0.8,
    top_p=0.95,
    max_tokens=100,
    stop_sequences=["\n\n"]
)
# Note: Claude doesn't support frequency/presence penalty
            

Local (Ollama / vLLM)

import requests

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3.2",
    "prompt": "Write a haiku about Python",
    "options": {
        "temperature": 0.8,
        "top_p": 0.95,
        "top_k": 40,
        "num_predict": 100,
        "stop": ["\n\n"]
    }
})
            

Debugging & Optimization Tips

• Outputs too repetitive? Increase frequency_penalty (0.3–0.7) or raise temperature slightly.
• Outputs too random/incoherent? Lower temperature (0.3–0.5) and use top_p=0.9.
• Need reproducibility? Set temperature=0 and use a fixed seed. Note: not 100% guaranteed across API versions.
• Generation stops early? Check for implicit stop sequences or increase max_tokens.
• Generation too long? Add explicit stop sequences or reduce max_tokens.
• Testing prompts? Start with temperature=0 for consistent debugging, then tune for production.

Related Topics

• Prompt Engineering
• Transformers Architecture
• Hallucinations & Grounding
• LLM Evaluation