The FormatCheckMetric validates that the actual output conforms to a specified format type.
How It Works
- Selects the appropriate validation rule based on
format_type - Applies the format-specific validation to the actual output
- Returns
1.0 if the output matches the format, 0.0 otherwise
| Format Type | Description | Example |
"email" | Valid email address | user@example.com |
"phone" | Phone number (digits, spaces, dashes, parentheses, optional +) | +1 (555) 123-4567 |
"date" | ISO 8601 date format | 2025-12-31 |
"url" | Valid URL with scheme | https://example.com |
Parameters
| Parameter | Type | Default | Description |
threshold | float | 1.0 | Minimum score to pass |
format_type | str | required | Format to validate: "email", "phone", "date", or "url" |
Required Fields
| Field | Required |
input | No |
actual_output | Yes |
expected_output | No |
Usage
from eval_lib.metrics import FormatCheckMetric
from eval_lib.test_case import EvalTestCase
import asyncio
test_case = EvalTestCase(
input="Generate a contact email",
actual_output="support@example.com"
)
metric = FormatCheckMetric(threshold=1.0, format_type="email")
result = asyncio.run(metric.evaluate(test_case))
print(result.score) # 1.0
Example Scenarios
Pass (1.0)
metric = FormatCheckMetric(format_type="url")
EvalTestCase(
input="Return the documentation URL",
actual_output="https://docs.example.com/api/v2"
)
# Valid URL with scheme
Fail (0.0)
metric = FormatCheckMetric(format_type="date")
EvalTestCase(
input="Return today's date",
actual_output="December 31, 2025"
)
# Not in ISO 8601 format (YYYY-MM-DD)