deepeval-evaluation

deepeval-evaluation

Overview

Per de-start, DeepEval is "an open-source LLM eval package" enabling "evaluation of LLM applications locally through test cases and metrics." The model: each test constructs an LLMTestCase, applies one or more Metric instances, and either asserts (assert_test) or batch-evaluates (evaluate). Pytest discovery + reporting works unchanged.

When to use

• The repo already uses pytest; LLM tests should live alongside unit tests.
• The user works with RAG and needs Faithfulness / Contextual-* metrics without writing them from scratch.
• The team wants a managed dashboard (Confident-AI) for regression tracking + prompt-vs-prompt comparison.
• Programmatic test-case authoring (data-driven from a CSV/JSONL) is needed and pytest fixtures fit better than YAML config.

Step 1 - Install

Per de-gh and de-start:

pip install -U deepeval

Optional Confident-AI login (for dashboard):

deepeval login

Per de-start: after login "Confident AI will generate testing reports and automate regression testing whenever you run a test run."

Step 2 - First test

Per de-gh (verbatim quickstart):

import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

def test_case():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
        retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
    )
    assert_test(test_case, [correctness_metric])

Run it:

deepeval test run test_chatbot.py

(Per de-gh.)

Step 3 - LLMTestCase fields

Per de-start, LLMTestCase fields:

Field	Required	Notes
input	yes	The user prompt / query
actual_output	yes	What the LLM produced
expected_output	optional	Reference answer (used by metrics that compare)
retrieval_context	optional	List of retrieved chunks for RAG metrics
context	optional	Ground-truth context for hallucination metric

Step 4 - Metric catalog

Per de-gh the available metrics include:

Metric	Use
GEval	Custom rubric-based scoring (LLM-as-judge with chain-of-thought)
AnswerRelevancyMetric	Does actual_output answer input?
FaithfulnessMetric	Does actual_output only state facts in retrieval_context?
ContextualRecallMetric	Does retrieval_context contain enough info to produce expected_output?
ContextualPrecisionMetric	Are relevant chunks ranked higher in retrieval_context?
ContextualRelevancyMetric	Are chunks in retrieval_context relevant to input?
HallucinationMetric	Does actual_output contradict context?
BiasMetric	Bias detection in actual_output
ToxicityMetric	Toxic-content detection
SummarizationMetric	Summary quality vs source
JsonCorrectnessMetric	Valid + schema-conformant JSON output

Each metric takes a threshold parameter; the test passes if score ≥ threshold.

Step 5 - Custom GEval pattern

GEval is the universal escape hatch when no built-in metric fits:

from deepeval.metrics import GEval
from deepeval.test_case import SingleTurnParams

professionalism = GEval(
    name="Professionalism",
    criteria="Determine if the response uses professional language without slang or contractions.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
    threshold=0.7,
)

The criteria string is the rubric; the judge model evaluates and returns a 0 - 1 score with reasoning attached.

Step 6 - Batch evaluation (no pytest)

For dataset-driven runs without pytest:

from deepeval import evaluate

evaluate(test_cases=[case1, case2, case3], metrics=[g_eval, faithfulness])

Returns scores per metric per case; useful for regression sweeps across a CSV/JSONL of historical inputs.

Step 7 - CI integration

deepeval test run tests/llm/ --run-async --workers 4
# pytest exit code propagates: nonzero if any assert_test fails

Combine with Confident-AI for a dashboard view of run history; Confident-AI is the company behind DeepEval (per de-gh).

Anti-patterns

Anti-pattern	Why it fails	Fix
threshold=0.0 on every metric	Tests never fail; eval theater	Pick real thresholds (0.5 - 0.8 typical)
Hallucination metric without context	Metric has nothing to compare against	Always pass context (Step 3)
Faithfulness metric without retrieval_context	Same problem	Pass retrieval chunks (Step 3)
Custom GEval criteria too vague	Judge produces inconsistent scores	Concrete criteria with examples (Step 5)
Skip --workers in CI	Sequential runs slow + costly	--workers 4 parallelization (Step 7)

Limitations

• LLM-as-judge metrics depend on judge-model quality + cost; pin judge model version in CI.
• Confident-AI is the managed dashboard; without it, regression tracking is manual (parse pytest output).
• Test cases live in Python files - not as discoverable as YAML configs for non-Python teammates (vs Promptfoo).
• Faithfulness / Contextual-* metrics need RAG-shaped data; for pure prompt evals, promptfoo-evaluation is lower-friction.

References

• de-gh - repository, install, quickstart, metric list
• de-start - getting-started overview
• promptfoo-evaluation, ragas-evaluation - sister tools (Promptfoo for YAML-config; Ragas for deeper RAG metric variety)
• prompt-eval-reviewer - adversarial reviewer