Helicone

Proxy-based LLM Observability Platform

Overview

Helicone is an open-source LLM observability platform that acts as a proxy to monitor, analyze, and optimize AI applications. With a single line of code, you get automatic logging of all LLM requests.

Built on Cloudflare's global network for high performance and reliability, Helicone provides real-time monitoring, cost optimization, and security features without impacting your application's latency.

Key Features

One-line Integration

Change your base URL to Helicone's proxy and instantly get full request/response logging.

Cost Tracking

Monitor spend, token usage, and latency. Set alerts for cost thresholds and track by project/user.

Caching & Rate Limiting

Cache responses to reduce latency and costs. Rate limit API usage to prevent overages.

Security Features

Built-in prompt injection prevention and malicious response protection powered by Meta's security models.

Implementation

Quick Setup

# Install Helicone
pip install helicone

# Set your API key
export HELICONE_API_KEY="your-api-key-here"

OpenAI Integration

from openai import OpenAI

# Just change the base URL
client = OpenAI(
    api_key="your-openai-key",
    base_url="https://oai.helicone.ai/v1"  # Add Helicone proxy
)

# Your existing code works unchanged
response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {"role": "user", "content": "Hello, world!"}
    ]
)

Advanced Configuration

import helicone

# Custom headers for better tracking
helicone_client = helicone.Helicone(
    api_key="your-api-key",
    headers={
        "Helicone-User-Id": "user-123",
        "Helicone-Session-Name": "chat-session-1",
        "Helicone-Property-Feature": "premium-chat",
        "Helicone-Cache-Enabled": "true"
    }
)

# Enable response caching
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Explain quantum computing"}],
    extra_headers={
        "Helicone-Cache-Enabled": "true",
        "Helicone-Cache-TTL": "3600"  # Cache for 1 hour
    }
)

Real-World Use Cases

SaaS Applications

Track usage per customer, implement cost controls, and optimize response times.

• Per-customer cost attribution
• Rate limiting by tier
• Performance analytics

AI-powered Startups

Monitor LLM costs, optimize prompts, and scale efficiently.

• Real-time cost monitoring
• Prompt optimization insights
• A/B testing framework

Enterprise Teams

Enterprise-grade monitoring with security and compliance features.

• SOC 2 compliance ready
• Custom retention policies
• Advanced threat detection

Research Projects

Track experimental prompts and measure model performance.

• Prompt experimentation
• Performance benchmarking
• Dataset tracking

Streaming Metrics

Track streaming-specific performance:

Time to First Token (TTFT)
Total streaming duration
Tokens per second throughput

Streaming Example

# Streaming with Helicone tracking
stream = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Tell me a story"}],
    stream=True,
    extra_headers={
        "Helicone-Stream-Enabled": "true"
    }
)

for chunk in stream:
    print(chunk.choices[0].delta.content, end="")

# Helicone automatically tracks TTFT and throughput

Quick Info

Type: Open-source
Architecture: Proxy-based
Infra: Cloudflare Edge
Deployment: Cloud/Self-hosted

Supported Providers

OpenAI Azure Anthropic Gemini Groq Cohere