Skip to content

Goal Achievement Rate

The Goal Achievement Rate metric evaluates whether the user actually got what they wanted from the conversation. Unlike Task Success Rate, which checks a formal task checklist, this metric looks at the outcome through a satisfaction lens — did the user walk away having what they came for?

It inspects positive and negative satisfaction signals (thanks, frustration, repeated clarifications, unmet requests) and grades the final outcome.

How It Works

  1. Goal Resolution — uses the user-provided user_goal if supplied, otherwise an LLM infers the desired outcome from the dialogue
  2. Signal Extraction — extracts positive_signals, negative_signals, and unmet_requests from the conversation
  3. Outcome Verdict — grades the outcome against the goal on a 5-level scale (fully/mostly/partial/minor/none)
  4. Summary — produces a 1–2 sentence assessment

The final score is the verdict weight penalized by 0.1 × len(negative_signals), floored at 0.

Parameters

Parameter Type Default Description
model str required LLM model ("gpt-4o", "anthropic:claude-3-5-sonnet-latest", "google:gemini-2.0-flash", "ollama:llama3", or CustomLLMClient)
threshold float 0.7 Minimum score to pass
verbose bool False Enable colored console output
user_goal str \| None None Caller-provided desired outcome. When set, skips LLM goal inference.

Required Fields

Works only with ConversationalEvalTestCase.

Field Required
turns (each with input and actual_output) Yes

Usage

from eval_lib import (
    GoalAchievementRateMetric,
    ConversationalEvalTestCase,
    EvalTestCase,
    evaluate_conversations,
)
import asyncio

metric = GoalAchievementRateMetric(
    model="gpt-4o",
    threshold=0.7,
    user_goal="Get a working fix for a Python import error",
)

conversation = ConversationalEvalTestCase(
    turns=[
        EvalTestCase(
            input="My script crashes with ModuleNotFoundError: requests",
            actual_output="Run `pip install requests` and re-run your script.",
        ),
        EvalTestCase(
            input="That worked, thanks!",
            actual_output="Glad it helped.",
        ),
    ]
)

results = asyncio.run(evaluate_conversations([conversation], [metric]))

Cost

  • With user_goal provided: 3 LLM API calls (signals, verdict, summary).
  • Without user_goal: 4 LLM API calls (the first one infers the goal).

Tips

  • Pair with Task Success Rate to distinguish "task was technically completed" from "user actually got what they wanted".
  • Use for support-style conversations where satisfaction signals are meaningful.
  • The negative_signals penalty (−0.1 per signal) means two frustration markers can drop a mostly (0.9) verdict below the default 0.7 threshold.