Skip to content

Contains

The ContainsMetric checks whether the actual output contains specified keywords, with configurable matching modes.

How It Works

  1. Optionally converts the output to lowercase for case-insensitive matching
  2. Checks each keyword against the output
  3. Applies the selected mode (any, all, or none) to determine the result
  4. Returns 1.0 if the condition is satisfied, 0.0 otherwise

Parameters

Parameter Type Default Description
threshold float 1.0 Minimum score to pass
keywords list[str] required List of keywords to check for
mode str "any" Matching mode: "any" (at least one), "all" (every keyword), "none" (no keywords)
case_sensitive bool True Whether matching is case-sensitive

Required Fields

Field Required
input No
actual_output Yes
expected_output No

Usage

from eval_lib.metrics import ContainsMetric
from eval_lib.test_case import EvalTestCase
import asyncio

test_case = EvalTestCase(
    input="List the primary colors",
    actual_output="The primary colors are red, blue, and yellow."
)

metric = ContainsMetric(
    threshold=1.0,
    keywords=["red", "blue", "yellow"],
    mode="all",
    case_sensitive=False
)
result = asyncio.run(metric.evaluate(test_case))
print(result.score)  # 1.0

Example Scenarios

Pass (1.0) — mode="any"

metric = ContainsMetric(keywords=["Python", "JavaScript"], mode="any")
EvalTestCase(
    input="Name a programming language",
    actual_output="Python is a popular language."
)
# At least one keyword found

Pass (1.0) — mode="none"

metric = ContainsMetric(keywords=["error", "fail", "exception"], mode="none")
EvalTestCase(
    input="Generate a success message",
    actual_output="Operation completed successfully."
)
# No forbidden keywords found

Fail (0.0)

metric = ContainsMetric(keywords=["red", "blue", "yellow"], mode="all")
EvalTestCase(
    input="List the primary colors",
    actual_output="The primary colors are red and blue."
)
# "yellow" is missing