Skip to content

Deterministic Metrics

Deterministic metrics are rule-based evaluation metrics that produce binary pass/fail scores (1.0 or 0.0) without requiring an LLM model. They are fast, predictable, and ideal for validating structured outputs.

Available Metrics

Metric Description
ExactMatchMetric Exact string comparison between actual and expected output
ContainsMetric Checks whether output contains specified keywords
StartsWithMetric Checks whether output starts with a given prefix
EndsWithMetric Checks whether output ends with a given suffix
RegexMatchMetric Matches output against a regular expression pattern
JsonSchemaMetric Validates output against a JSON schema
FormatCheckMetric Validates output format (email, phone, date, URL)
LengthCheckMetric Validates output length in characters or words
LanguageDetectionMetric Detects and validates the language of the output
NonEmptyMetric Checks that the output is not empty

Key Properties

  • No LLM required — all metrics are computed locally using deterministic rules
  • Binary scoring — every metric returns either 1.0 (pass) or 0.0 (fail)
  • Fast execution — no network calls or model inference needed
  • Import path — all metrics are available from eval_lib.metrics.deterministic_metrics

Usage

from eval_lib.metrics import ExactMatchMetric, ContainsMetric
from eval_lib.test_case import EvalTestCase
import asyncio

test_case = EvalTestCase(
    input="What is 2 + 2?",
    actual_output="4",
    expected_output="4"
)

metrics = [
    ExactMatchMetric(threshold=1.0),
    ContainsMetric(threshold=1.0, keywords=["4"], mode="any"),
]

for metric in metrics:
    result = asyncio.run(metric.evaluate(test_case))
    print(f"{metric.__class__.__name__}: {result.score}")