Skip to content

Repetitive Pattern Detection

The Repetitive Pattern Detection metric detects whether the agent gets stuck in a loop — repeating the same phrases, tool calls, or responses without making progress.

Higher score = fewer repetitions = better. The verdict scale is inverted compared to peer metrics so that "higher is better" semantics stay consistent.

How It Works

  1. Single verdict — one LLM call analyzes the whole dialogue and returns {verdict, repeated_segments, reason}
  2. Summary — concise assessment of the repetition severity

The verdict maps to a score via inverted weights:

Verdict Score Meaning
none 1.0 No repetition; every turn adds new value
minor 0.9 A single slightly-repeated phrase but conversation progresses
partial 0.7 Noticeable repetition but still some progress
mostly 0.3 Frequent repetition, little progress
fully 0.0 Stuck in a loop; same action/response cycles without any progress

Parameters

Parameter Type Default Description
model str required LLM model ("gpt-4o", "anthropic:claude-3-5-sonnet-latest", "google:gemini-2.0-flash", "ollama:llama3", or CustomLLMClient)
threshold float 0.7 Minimum score to pass (i.e. "no more than partial repetition")
verbose bool False Enable colored console output

Required Fields

Works only with ConversationalEvalTestCase.

Field Required
turns (each with input and actual_output) Yes

Usage

from eval_lib import (
    RepetitivePatternDetectionMetric,
    ConversationalEvalTestCase,
    EvalTestCase,
    evaluate_conversations,
)
import asyncio

metric = RepetitivePatternDetectionMetric(model="gpt-4o", threshold=0.7)

conversation = ConversationalEvalTestCase(
    turns=[
        EvalTestCase(
            input="Can you help me with X?",
            actual_output="I cannot help with that request.",
        ),
        EvalTestCase(
            input="Please, it's urgent",
            actual_output="I cannot help with that request.",
        ),
        EvalTestCase(
            input="At least point me somewhere",
            actual_output="I cannot help with that request.",
        ),
    ]
)

results = asyncio.run(evaluate_conversations([conversation], [metric]))
# score will be 0.0 — the agent is fully stuck in a refusal loop

Cost

1 LLM API call per evaluation (single verdict pass returns the reason directly).

Tips

  • Inspect evaluation_log["repeated_segments"] to see exactly which phrases or actions the LLM identified as repetitive.
  • Combine with Conversational Flow Rate — flow can be broken for reasons other than repetition (e.g. ignoring user intent), so the two metrics cover different failure modes.
  • Pair with Tool Error Detection for tool-using agents: a repeated_failure in tool calls often surfaces as a loop here too.