Task Success Rate¶

The Task Success Rate metric evaluates whether the AI assistant successfully helped the user achieve their goal. Works with both single-turn and multi-turn conversations.

How It Works¶

Goal Inference — analyzes the dialogue to understand the user's goal (skipped when task_description is provided)
Criteria Generation — generates specific success criteria (skipped when success_criteria is provided)
Verdict Generation — evaluates each criterion using the 5-level scale
Score Aggregation — combines verdicts using temperature-controlled softmax

When both task_description and success_criteria are supplied the metric makes only 2 LLM calls instead of 4, and the evaluation becomes fully deterministic with respect to what "success" means.

Parameters¶

Parameter	Type	Default	Description
`model`	`str`	required	LLM model (`"gpt-4o"`, `"anthropic:claude-3-5-sonnet-latest"`, `"google:gemini-2.0-flash"`, `"ollama:llama3"`, or `CustomLLMClient`)
`threshold`	`float`	`0.7`	Minimum score to pass
`temperature`	`float`	`0.5`	Aggregation strictness
`verbose`	`bool`	`False`	Print LLM prompts/responses for debugging
`task_description`	`str \\| None`	`None`	Caller-provided task description. When set, skips LLM goal inference and uses this value as the user goal.
`success_criteria`	`list[str] \\| None`	`None`	Caller-provided explicit list of success criteria. When set, skips LLM criteria generation.

Required Fields¶

Field	Required
`input`	Yes
`actual_output`	Yes

Usage¶

Single-Turn¶

from eval_lib import TaskSuccessRateMetric, EvalTestCase, evaluate
import asyncio

test_case = EvalTestCase(
    input="Help me write a regex to match email addresses.",
    actual_output="Here's a regex for email matching: `[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}`. This handles most standard email formats."
)

metric = TaskSuccessRateMetric(model="gpt-4o", threshold=0.7)
results = asyncio.run(evaluate([test_case], [metric]))

Multi-Turn (Conversational)¶

from eval_lib import (
    ConversationalEvalTestCase,
    EvalTestCase,
    evaluate_conversations,
    TaskSuccessRateMetric,
)

conversation = ConversationalEvalTestCase(
    turns=[
        EvalTestCase(
            input="I need to set up a CI/CD pipeline for my Node.js project.",
            actual_output="I can help with that! Are you using GitHub Actions, GitLab CI, or another platform?"
        ),
        EvalTestCase(
            input="GitHub Actions.",
            actual_output="Here's a workflow file for your Node.js project: [provides complete .github/workflows/ci.yml with test, lint, and deploy steps]"
        ),
        EvalTestCase(
            input="Can you add caching for node_modules?",
            actual_output="Sure! Add this caching step: [provides cache action configuration with hash-based key]"
        ),
    ]
)

metric = TaskSuccessRateMetric(model="gpt-4o", threshold=0.7)
results = asyncio.run(evaluate_conversations([conversation], [metric]))

User-Provided Task and Success Criteria¶

When you already know what the task is and what counts as success, pass them directly to skip the goal-inference and criteria-generation LLM calls. This is cheaper, faster, and more deterministic:

from eval_lib import TaskSuccessRateMetric, ConversationalEvalTestCase, EvalTestCase, evaluate_conversations
import asyncio

metric = TaskSuccessRateMetric(
    model="gpt-4o",
    threshold=0.7,
    task_description="Book a round-trip flight to Paris for next Monday",
    success_criteria=[
        "Assistant presented bookable flight options with prices",
        "Assistant confirmed the booking with dates",
    ],
)

conversation = ConversationalEvalTestCase(
    turns=[
        EvalTestCase(
            input="Book me a flight to Paris for next Monday",
            actual_output="Here are three options: [lists flights with prices]",
        ),
        EvalTestCase(
            input="Take the 9am one",
            actual_output="Booked. Confirmation #AB123, departing next Monday at 9am.",
        ),
    ]
)

results = asyncio.run(evaluate_conversations([conversation], [metric]))

The evaluation_log will include task_description_source and success_criteria_source fields ("user_provided" or "llm_inferred") so you can see exactly which path was taken.

Cost¶

Default: 4 LLM API calls per evaluation (goal inference, criteria generation, verdicts, summary).
With task_description only: 3 LLM calls (goal inference skipped).
With both task_description and success_criteria: 2 LLM calls (only verdicts and summary).

Tips¶

Use with Tool Correctness for agents that use tools to complete tasks
For multi-turn conversations, the metric considers the entire dialogue trajectory
Set lower threshold (0.5) for complex, open-ended tasks