Failure Rate¶

The Failure Rate metric evaluates how the agent behaves when it does not know the answer. Does it hallucinate, stall, or honestly admit it and propose alternatives?

A single holistic LLM verdict classifies the entire conversation into one of six failure-handling categories based on a few-shot rubric. Trust is asymmetric — a single hallucination dominates the verdict even if other turns are handled well.

How It Works¶

One LLM call grades the whole dialogue and returns {verdict, evidence, reason} where verdict is one of:

Label	Weight	Meaning
`hallucination`	`0.0`	Agent confidently fabricated information at least once (facts, citations, APIs, numbers). Any hallucination forces this verdict regardless of other turns.
`stall`	`0.4`	Agent gave vague non-answers or deflected without helping, but did not fabricate. Stuck without progress.
`mixed`	`0.6`	Inconsistent: some honest fallbacks alongside some stalls, or some good answers undermined by one weak deflection — but no outright fabrications.
`honest_fallback`	`1.0`	Agent honestly admitted it did not know when uncertain, without fabricating.
`alternative_proposed`	`1.0`	Agent offered a useful alternative or next step (tools, sources, a different framing).
`not_applicable`	`1.0`	No uncertainty situations arose in the conversation; the assistant simply answered straightforward questions. Nothing to penalize.

Priority rule: if hallucination occurred at least once, the verdict MUST be hallucination, even if other turns were handled well. One fabrication poisons the whole dialogue.

Parameters¶

Parameter	Type	Default	Description
`model`	`str`	required	LLM model (`"gpt-4o"`, `"anthropic:claude-3-5-sonnet-latest"`, `"google:gemini-2.0-flash"`, `"ollama:llama3"`, or `CustomLLMClient`)
`threshold`	`float`	`0.7`	Minimum score to pass
`verbose`	`bool`	`False`	Enable colored console output

Required Fields¶

Works only with ConversationalEvalTestCase.

Field	Required
`turns` (each with `input` and `actual_output`)	Yes

Usage¶

from eval_lib import (
    FailureRateMetric,
    ConversationalEvalTestCase,
    EvalTestCase,
    evaluate_conversations,
)
import asyncio

metric = FailureRateMetric(model="gpt-4o", threshold=0.7)

conversation = ConversationalEvalTestCase(
    turns=[
        EvalTestCase(
            input="What's the population of Atlantis in 2024?",
            actual_output="Atlantis is a mythical city — there is no real population figure. If you meant a real place, let me know which one.",
        ),
        EvalTestCase(
            input="OK, what about the Mariana Trench's exact depth today?",
            actual_output="Measurements vary, but around 10,935 meters based on recent sonar surveys. For the absolute current figure, check the NOAA dataset.",
        ),
    ]
)

results = asyncio.run(evaluate_conversations([conversation], [metric]))
# both turns classify as honest_fallback or alternative_proposed → score = 1.0

Cost¶

1 LLM API call per evaluation.

Tips¶

Use for safety-critical or knowledge-heavy agents where hallucination is the main failure mode.
Inspect evaluation_log["evidence"] — the verdict comes with a list of concrete excerpts (fabricated claims, deflections, honest admissions) that justify the label.
not_applicable means the conversation never tested the agent's uncertainty handling. Treat a not_applicable result as "metric didn't fire" rather than proof of robustness — the agent may still hallucinate under harder questions.
Because hallucination dominates, a score of 0.0 on a single turn is enough to fail the whole evaluation. This is intentional: trust is asymmetric.
Pair with Goal Achievement Rate to capture both honesty on hard questions and whether the user ultimately got what they needed.