Conversational Flow Rate¶

The Conversational Flow Rate metric grades how natural and coherent a multi-turn dialogue is. It penalizes unnecessary clarifications, redundant questions, ignored user signals, and irrelevant tangents.

How It Works¶

A single holistic LLM call grades the dialogue as a whole against a few-shot rubric and returns one verdict on the 5-level scale (fully/mostly/partial/minor/none).

Flow is judged as a property of the entire sequence, not an average of per-turn scores. A single severely broken turn that derails the rest of the conversation pulls the verdict down even if other turns are fine. Later turns are weighted more heavily than earlier ones when they diverge — the end state matters most.

The rubric explicitly considers whether the assistant:

• directly addresses user signals,
• avoids redundant clarifications,
• stays coherent with prior turns,
• and avoids irrelevant tangents.

The verdict maps to a score:

Parameters¶

Required Fields¶

Works only with ConversationalEvalTestCase.

Usage¶

from eval_lib import (
    ConversationalFlowRateMetric,
    ConversationalEvalTestCase,
    EvalTestCase,
    evaluate_conversations,
)
import asyncio

metric = ConversationalFlowRateMetric(model="gpt-4o", threshold=0.7)

conversation = ConversationalEvalTestCase(
    turns=[
        EvalTestCase(
            input="I need to reset my password",
            actual_output="I can help. Please confirm the email associated with your account.",
        ),
        EvalTestCase(
            input="user@example.com",
            actual_output="Password reset link sent to user@example.com. Check your inbox.",
        ),
    ]
)

results = asyncio.run(evaluate_conversations([conversation], [metric]))

Cost¶

1 LLM API call per evaluation.

Tips¶

• Pair with Repetitive Pattern Detection to catch loops that also count as broken flow, but for different reasons.
• Inspect evaluation_log["key_issues"] — the holistic verdict comes with a list of the main problems the LLM identified, which is usually where you should start fixing the agent.
• Because the verdict is holistic, a single severely broken turn is enough to drop the score to none; use this metric when you want to catch catastrophic flow breakdowns rather than measure average turn quality.