DeepEval

The Open-Source LLM Evaluation Framework — "Pytest for LLMs"

What is DeepEval?

DeepEval is an open-source evaluation framework for LLM applications, built by Confident AI. It lets you "unit test" LLM outputs the same way you test regular code — but using research-backed metrics designed for non-deterministic, generative systems. It integrates natively with pytest, runs evals locally, and ships with 14+ ready-to-use metrics for RAG, agents, and conversational use cases.

"Think of DeepEval as Pytest for LLMs. You write test cases, pick the metrics that matter, and get pass/fail results in CI — so you can ship LLM features with the same confidence as any other code."

— Confident AI

Pytest-Native

CI-ready tests

14+ Metrics

RAG, agents, safety

Any Model

OpenAI, local, custom

Open Source

Runs locally

Core Concepts

Test Case

An LLMTestCase captures one interaction: the input, the model's actual_output, an optional expected_output, the retrieval_context (for RAG), and context. Metrics are evaluated against this object.

Metric

A scorer with a threshold. Each metric produces a score (0–1) plus a reason. Many DeepEval metrics are LLM-as-judge based and self-explaining, so you get a natural-language rationale for each pass/fail.

Dataset

An EvaluationDataset is a collection of test cases (goldens). You can author them by hand, pull them from CSV/JSON, or auto-generate synthetic ones with the built-in Synthesizer.

assert_test / evaluate

Use assert_test() inside pytest for CI gating, or evaluate() for bulk runs and reports. Run from the CLI with deepeval test run.

Key Metrics

Metric	What It Measures	Use Case
G-Eval	Custom LLM-judge on any criteria you define	Tone, correctness, custom rules
Answer Relevancy	Is the output relevant to the input?	Q&A, chatbots
Faithfulness	Output grounded in retrieval context	RAG hallucination check
Contextual Precision / Recall	Quality & coverage of retrieved chunks	RAG retriever tuning
Hallucination	Contradictions vs. provided context	Factuality
Toxicity / Bias	Harmful or biased content	Safety evaluation
Task Completion / Tool Correctness	Did the agent achieve the goal & call right tools	Agent evaluation
Summarization	Alignment & coverage of a summary	Summarization tasks

Quick Start

# Install & set your judge model key
pip install -U deepeval
export OPENAI_API_KEY="sk-..."

# test_rag.py  — run with: deepeval test run test_rag.py
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric, FaithfulnessMetric

def test_rag_answer():
    test_case = LLMTestCase(
        input="What is the capital of France?",
        actual_output="The capital of France is Paris.",
        retrieval_context=["France is a country in Europe. Its capital is Paris."],
    )

    relevancy = AnswerRelevancyMetric(threshold=0.7)
    faithfulness = FaithfulnessMetric(threshold=0.7)

    # Fails the test (and CI) if either metric is below threshold
    assert_test(test_case, [relevancy, faithfulness])

Custom Criteria with G-Eval

When no built-in metric fits, G-Eval lets you define a metric in plain English. It uses chain-of-thought LLM judging under the hood.

from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, LLMTestCaseParams

correctness = GEval(
    name="Correctness",
    criteria="Determine if the actual output is factually correct based on the expected output.",
    evaluation_params=[
        LLMTestCaseParams.ACTUAL_OUTPUT,
        LLMTestCaseParams.EXPECTED_OUTPUT,
    ],
    threshold=0.5,
)

test_case = LLMTestCase(
    input="How many planets are in the solar system?",
    actual_output="There are 8 planets.",
    expected_output="8",
)

correctness.measure(test_case)
print(correctness.score, correctness.reason)

Where DeepEval Fits

Dataset / Goldens

Run LLM App

DeepEval Metrics

CI Gate

Confident AI Dashboard

Best Practices

Do This

Gate deployments with deepeval test run in CI
Pick RAG-specific metrics for RAG apps (faithfulness, contextual recall)
Use a strong judge model (e.g. GPT-4 class) for reliable scores
Version your datasets and track scores over time
Define custom rules with G-Eval when built-ins don't fit

Avoid This

Judging with a weak/cheap model and trusting the scores blindly
Tiny test sets that miss edge cases
Setting thresholds without calibrating against human judgment
Ignoring metric cost/latency when running large suites
Treating evals as one-time instead of continuous

Resources

GitHub Repository

Source code, examples, and the full metric catalogue.

github.com/confident-ai/deepeval →

Documentation

Official docs, metric reference, and the Confident AI platform.

docs.confident-ai.com →

The Evaluation Tool Landscape

DeepEval is one of several complementary tools. A mature eval stack often combines a CI test runner, a RAG-specific scorer, and an observability platform that re-runs metrics on live traffic.

Tool	Sweet Spot	Type
DeepEval	Pytest-style unit tests for LLM apps & agents	Library / CI
Ragas	Reference-free RAG metrics (faithfulness, recall)	Library
Promptfoo	YAML model comparison & AI red teaming	CLI / CI
Opik	Trace + eval + production monitoring	Platform
Langfuse	Open-source tracing, datasets & online LLM-judge	Platform
LangSmith	LangChain-native dev, tracing & eval	Platform