GenAIHub
EN | PT
← Back to Technical Section

W&B Weave

LLM Application Development Toolkit from Weights & Biases

What is W&B Weave?

W&B Weave is a lightweight, open-source toolkit for developing, debugging, and evaluating LLM applications. Built by Weights & Biases, the company behind the popular ML experiment tracking platform, Weave extends their expertise to generative AI workflows.

Unlike traditional W&B which focuses on model training, Weave is purpose-built for LLM application development—tracing prompts, evaluating outputs, versioning datasets, and comparing model performance in production-like scenarios.

Apache
Open Source
W&B
Integration
10+
Built-in Scorers
@weave.op
Auto-Tracing

Core Capabilities

Automatic Tracing with @weave.op

Core

Add @weave.op() to any function and Weave automatically logs inputs, outputs, execution time, and exceptions. Works with sync and async functions.

Nested Traces Auto-versioning Latency Tracking Cost Estimation

Evaluation Framework

weave.Evaluation

Run structured evaluations against datasets with built-in scorers or custom metrics. Compare results across models, prompts, and configurations in the W&B dashboard.

Dataset Management Leaderboards Visual Comparison Custom Scorers

weave.Model Class

Versioned

Wrap your LLM applications in versioned, trackable objects. Every change to prompts, parameters, or logic creates a new version that can be compared and rolled back.

Auto-versioning Lineage Tracking Reproducibility Configuration

Built-in Scorers

10+ Metrics

Pre-built scorers for common evaluation needs: hallucination detection, relevance, summarization quality, toxicity, and more. Easily extend with custom LLM-as-a-Judge scorers.

Hallucination Relevance Faithfulness Toxicity

Serve & Deploy

Production

Deploy traced functions as API endpoints with a single command. All calls remain traced and visible in the W&B dashboard for production monitoring.

One-line Deploy Production Tracing API Endpoint Feedback Loop

Quick Start

Terminal - Installation
# Install Weave
pip install weave

# Login to W&B (stores API key)
wandb login
Python - Basic Tracing with @weave.op
import weave
from openai import OpenAI

# Initialize Weave project
weave.init("my-llm-project")

client = OpenAI()

@weave.op()
def generate_response(prompt: str) -> str:
    """This function is automatically traced."""
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

# Every call is logged to W&B
result = generate_response("Explain quantum computing")
print(result)

# View traces at: https://wandb.ai/{entity}/{project}/weave

Creating a Versioned Model

Python - weave.Model Class
import weave
from openai import OpenAI

weave.init("rag-chatbot")

class RAGChatbot(weave.Model):
    """Versioned RAG chatbot - changes auto-create new versions."""
    
    model_name: str = "gpt-4"
    temperature: float = 0.7
    system_prompt: str = "You are a helpful assistant."
    
    @weave.op()
    def predict(self, question: str, context: str) -> str:
        client = OpenAI()
        response = client.chat.completions.create(
            model=self.model_name,
            temperature=self.temperature,
            messages=[
                {"role": "system", "content": self.system_prompt},
                {"role": "user", "content": f"Context: {context}\n\nQuestion: {question}"}
            ]
        )
        return response.choices[0].message.content

# Create and use model
chatbot = RAGChatbot(temperature=0.5)
answer = chatbot.predict(
    question="What is Weave?",
    context="Weave is an LLM development toolkit."
)

Running Evaluations

Python - Evaluation with Scorers
import weave
from weave.scorers import HallucinationScorer, RelevanceScorer

weave.init("evaluation-demo")

# Create evaluation dataset
dataset = [
    {
        "question": "What is the capital of France?",
        "context": "France is a country in Europe. Paris is its capital city.",
        "expected": "Paris"
    },
    {
        "question": "What language is spoken in France?",
        "context": "French is the official language of France.",
        "expected": "French"
    }
]

# Define scorers
scorers = [
    HallucinationScorer(),
    RelevanceScorer(),
]

# Run evaluation
evaluation = weave.Evaluation(
    dataset=dataset,
    scorers=scorers
)

# Evaluate your model
results = evaluation.evaluate(chatbot)
print(f"Results: {results}")

# View detailed results in W&B dashboard

Available Scorers

Scorer Purpose Type
HallucinationScorer Detect factual errors not in context LLM-judge
RelevanceScorer Measure answer relevance to question LLM-judge
SummarizationScorer Evaluate summary quality LLM-judge
ToxicityScorer Detect harmful or offensive content Classification
ContextRelevanceScorer RAG context relevance to query LLM-judge
Custom Scorer Define your own evaluation logic Function

LLM Provider Support

OpenAI
Anthropic
Google Gemini
Hugging Face
Cohere
LangChain
LlamaIndex
DSPy

W&B Ecosystem Integration

Weave integrates seamlessly with the broader Weights & Biases platform:

W&B Experiments

Use same dashboard for ML training and LLM development

W&B Artifacts

Version datasets, models, and evaluation results

W&B Teams

Collaborate with shared projects and dashboards

Weave vs Alternatives

Feature W&B Weave Langfuse Phoenix
ML Lifecycle Integration Full W&B LLM only LLM only
Tracing Approach @weave.op decorator @observe decorator OpenTelemetry
Model Versioning weave.Model Limited No
Embedding Viz No No Advanced
Best For ML teams using W&B Production monitoring RAG debugging

Resources

Documentation

Official guides & tutorials

GitHub

Source code & examples

Product Page

Overview & pricing

Quick Start

Get started in 5 minutes

Related Topics

Langfuse Arize Phoenix LLMOps Observability LLM Evaluation RAG

Test Your Knowledge

Score 8/10 or higher to pass