Skip to content

Regex Match

The RegexMatchMetric matches the actual output against a regular expression pattern.

How It Works

  1. Compiles the provided regex pattern
  2. Applies either a full match or a partial search depending on the full_match parameter
  3. Returns 1.0 if the pattern matches, 0.0 otherwise

Parameters

Parameter Type Default Description
threshold float 1.0 Minimum score to pass
pattern str required Regular expression pattern
full_match bool False If True, the entire output must match the pattern. If False, a partial match (search) is sufficient

Required Fields

Field Required
input No
actual_output Yes
expected_output No

Usage

from eval_lib.metrics import RegexMatchMetric
from eval_lib.test_case import EvalTestCase
import asyncio

test_case = EvalTestCase(
    input="Generate a date",
    actual_output="2025-12-31"
)

metric = RegexMatchMetric(
    threshold=1.0,
    pattern=r"^\d{4}-\d{2}-\d{2}$",
    full_match=True
)
result = asyncio.run(metric.evaluate(test_case))
print(result.score)  # 1.0

Example Scenarios

Pass (1.0) — partial match

metric = RegexMatchMetric(pattern=r"\d+\.\d{2}", full_match=False)
EvalTestCase(
    input="What is the price?",
    actual_output="The total price is $19.99 including tax."
)
# Pattern found within the output

Pass (1.0) — full match

metric = RegexMatchMetric(pattern=r"[A-Z]{3}-\d{4}", full_match=True)
EvalTestCase(
    input="Generate a product code",
    actual_output="ABC-1234"
)
# Entire output matches the pattern

Fail (0.0)

metric = RegexMatchMetric(pattern=r"^\d+$", full_match=True)
EvalTestCase(
    input="Return a number",
    actual_output="The answer is 42"
)
# Full output does not match digits-only pattern