Custom Metrics¶

The CustomEvalMetric lets you define your own evaluation criteria without writing metric code. You pass a list of natural-language criteria that reference data via {{placeholders}}, pick a scoring strategy, and optionally repeat the judge several times for consensus voting.

How It Works¶

1. Data collection — the metric resolves every value addressable from the test case: input, actual_output, expected_output, retrieval_context, dataset row columns, system_prompt, and any extra_fields you passed on the EvalTestCase.
2. Criterion filtering — each criterion must reference at least one {{placeholder}}, and every placeholder it references must resolve. Criteria that fail either check are skipped and logged in evaluation_log["skipped_criteria"].
3. Prompt build — the metric renders a DATA block listing only the values that appear in some kept criterion, followed by the criteria list.
4. Judge call(s) — depending on strategy (below), the judge either verdicts each criterion or returns one overall 0-10 score. If n_runs > 1, the judge is called that many times in parallel.
5. Aggregation — per-run outputs are combined (majority / median / mean) and, for the verdict strategy, aggregated across criteria via TCVA.

Two Scoring Strategies¶

strategy="verdict" (default)¶

The judge returns one verdict per criterion, chosen from a five-level scale:

Verdict weights are aggregated into a single 0.0-1.0 score via TCVA (Temperature-Controlled Verdict Aggregation) — the same generalized power mean used by the built-in metrics.

Use this strategy when criteria are independent and you want granular per-criterion feedback in the evaluation log.

strategy="direct"¶

A single LLM-as-a-judge prompt asks the judge to rate how well the DATA satisfies all criteria together on an integer scale of 0-10, then the raw score is normalized to 0.0-1.0 (raw / 10).

Use this strategy for holistic assessments — brand voice, overall clarity, subjective quality — where separating criteria into independent verdicts is artificial.

Consensus over n_runs¶

Any strategy can be repeated n_runs times to reduce variance. When n_runs > 1:

• The judge is called n_runs times in parallel.
• The LLM sampling temperature is automatically raised (to 0.7) so runs actually differ; a deterministic judge would make consensus voting a no-op.
• Results are combined per aggregation:

For verdict runs, the aggregated per-criterion weights are then passed through TCVA as usual.

n_runs is capped at 15 (safety ceiling); the default 1 means no consensus and stays fully deterministic.

Parameters¶

Available {{placeholders}}¶

Placeholders are extracted at evaluate-time — the criteria strings themselves are reusable across test cases.

Usage¶

Basic — verdict strategy, single run¶

from eval_lib import CustomEvalMetric, EvalTestCase, evaluate
import asyncio

metric = CustomEvalMetric(
    model="gpt-4o",
    threshold=0.7,
    name="AnswerQuality",
    evaluation_criteria=[
        "{{actual_output}} directly answers {{input}}",
        "{{actual_output}} is factually grounded in {{retrieval_context}}",
        "{{actual_output}} is concise and free of filler",
    ],
)

test_case = EvalTestCase(
    input="Explain how garbage collection works in Python.",
    actual_output="Python uses reference counting as its primary garbage collection mechanism...",
    retrieval_context=[
        "Python's garbage collector combines reference counting with a generational cycle detector."
    ],
)

results = asyncio.run(evaluate([test_case], [metric]))

Direct strategy — single holistic score¶

metric = CustomEvalMetric(
    model="gpt-4o",
    threshold=0.7,
    name="BrandVoice",
    evaluation_criteria=[
        "{{actual_output}} sounds warm, confident, and helpful",
        "{{actual_output}} avoids jargon a first-time customer wouldn't understand",
        "{{actual_output}} matches the tone set by {{system_prompt}}",
    ],
    strategy="direct",
)

The judge returns one integer 0-10 for the whole thing; the metric normalizes it to 0.0-1.0.

Consensus voting — 5 runs, majority vote¶

metric = CustomEvalMetric(
    model="gpt-4o",
    threshold=0.7,
    name="MedicalInfoQuality",
    evaluation_criteria=[
        "{{actual_output}} contains no unsafe medical claims",
        "{{actual_output}} recommends consulting a healthcare professional",
        "{{actual_output}} is grounded in {{retrieval_context}}",
    ],
    strategy="verdict",
    n_runs=5,
    aggregation="majority",
    temperature=0.2,  # TCVA — strict aggregation across criteria
)

For each criterion the judge is called 5 times; the mode verdict label wins. Then TCVA at temperature=0.2 keeps aggregation strict — any single "none" or "minor" verdict drags the final score down.

Consensus + direct — 3 runs, median score¶

metric = CustomEvalMetric(
    model="anthropic:claude-sonnet-4-0",
    threshold=0.7,
    name="EducationalClarity",
    evaluation_criteria=[
        "{{actual_output}} uses concrete examples appropriate to {{input}}",
        "{{actual_output}} builds concepts in a progressive order",
        "{{actual_output}} defines every non-obvious term",
    ],
    strategy="direct",
    n_runs=3,
    aggregation="median",
)

Median across 3 runs of the 0-10 judge score, then normalized.

Using extra_fields for domain data¶

test_case = EvalTestCase(
    input="Summarize the article",
    actual_output="AI is transforming healthcare...",
    extra_fields={
        "follow_up_questions": ["Will AI replace doctors?", "How is it regulated?"],
        "target_length": 200,
    },
)

metric = CustomEvalMetric(
    model="gpt-4o",
    threshold=0.7,
    name="SummaryQuality",
    evaluation_criteria=[
        "{{actual_output}} is a fair summary of {{input}}",
        "Each item in {{follow_up_questions}} is relevant to {{actual_output}}",
        "{{actual_output}} is close in length to {{target_length}} words",
    ],
)

Result Fields¶

The metric returns the standard MetricPattern shape; the extra detail lives under evaluation_log:

Cost¶

• strategy="verdict": one LLM call per run (n_runs calls total).
• strategy="direct": one LLM call per run (n_runs calls total).

Runs are executed concurrently via asyncio.gather, so wall-clock time is the slowest single run, not the sum.

Practical Tips¶

1. Start with n_runs=1 to iterate on your criteria; enable consensus only once the criteria stabilize — it multiplies cost linearly.
2. strategy="verdict" is better for auditability — the log shows which criterion failed. strategy="direct" is better for subjective / holistic dimensions.
3. Use low TCVA temperature (0.1-0.3) for safety-critical evaluations (medical, legal, compliance) — any weak verdict drags the aggregate down.
4. Prefer median over mean for consensus — one outlier verdict from a stochastic judge shouldn't move the score much.
5. Combine with built-in metrics. Use FaithfulnessMetric for factual grounding, AnswerRelevancyMetric for topical fit, and CustomEvalMetric for domain- or brand-specific rules the built-in judges don't cover.