Why LLM Observability?
LLM applications are non-deterministic and complex. Unlike traditional software, the same input can produce different outputs, making debugging challenging. Observability provides the visibility needed to understand how and why your AI systems behave the way they do.
"LLM observability involves collecting and correlating logs, real-time evaluation metrics, and traces to understand the context of unexpected outputs or errors. It provides full visibility into the entire LLM pipeline, from prompt to response."
Metrics
Latency, throughput, token usage, error rates
Traces
End-to-end request flow visualization
Logs
Prompts, responses, errors, events
Key Concepts
LLM Monitoring
Continuously tracks LLM performance in production. Focuses on metrics like response time, error rates, usage patterns, and cost to ensure expected behavior and identify performance degradation.
LLM Tracing
Captures the flow of requests through your LLM pipeline, providing a timeline of events. Essential for diagnosing issues in complex workflows like RAG systems or multi-step agent reasoning.
Evaluation
Systematic assessment of LLM outputs using automated metrics, LLM-as-a-judge, or human feedback. Includes techniques like golden datasets, regression testing, and A/B testing.
Essential Metrics to Track
| Category | Metric | Why It Matters |
|---|---|---|
| Performance | Latency (p50, p95, p99) | User experience, SLA compliance |
| Cost | Token usage (input/output) | Budget control, optimization |
| Reliability | Error rate, success rate | System health, availability |
| Quality | Hallucination rate | Output accuracy, trust |
| Agent | Tool invocation rate | Agent behavior analysis |
| Resources | CPU/Memory usage | Infrastructure scaling |
Observability Tools Comparison
LangSmith
by LangChain- Deep LangChain/LangGraph integration
- Comprehensive debugging & testing
- Built-in evaluators (LLM-as-judge)
- Closed-source, paid for self-hosting
LangFuse
Open Source- Open-source, free self-hosting
- Framework-agnostic (any LLM)
- Prompt management & datasets
- Multi-modal support
Weights & Biases
Weave- MLOps + LLMOps unified
- Experiment tracking
- Strong visualization
- Learning curve for new users
OpenTelemetry
GenAI Semantic Conventions- Vendor-neutral standard
- Works with any backend
- AI-specific semantic conventions
- Requires more setup
Other Notable Tools
OpenTelemetry for LLM Observability
"OpenTelemetry provides a standardized, vendor-neutral framework for collecting telemetry data. It transforms opaque AI 'black boxes' into debuggable and auditable systems."
Key Benefits
Vendor Neutral
Works with Datadog, Grafana, Jaeger, etc.
Single SDK
Collect metrics, traces, logs together
Semantic Conventions
Standardized AI/LLM attributes
Distributed Tracing
Multi-agent system visibility
Best Practices
Do This
- Instrument from day one, not retrofit
- Track all agent steps as spans
- Include prompts and responses in traces
- Use semantic conventions for AI
- Correlate logs with trace IDs
- Monitor token usage for cost control
Avoid This
- Logging only final outputs
- Ignoring intermediate reasoning steps
- Over-sampling in high-volume systems
- Storing PII in traces without masking
- Separate tools for metrics/traces/logs
- Waiting for production issues to add monitoring
Quick Start: LangFuse with Python
# Install
pip install langfuse openai
# Initialize LangFuse
from langfuse.decorators import observe
from langfuse.openai import openai
# Auto-trace OpenAI calls
@observe()
def get_answer(question: str) -> str:
response = openai.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": question}]
)
return response.choices[0].message.content
# All calls are now traced automatically!
answer = get_answer("What is observability?")See full documentation: langfuse.com/docs
Research & References
LangFuse Documentation
Open-source LLM observability
LangSmith Documentation
LangChain's observability platform
OpenTelemetry GenAI
AI semantic conventions
LLM Observability Guide
Confident AI comprehensive guide
OpenLLMetry
OpenTelemetry for LLM apps
LangChain State of AI 2024
Industry trends and insights