Answer Precision¶

The Answer Precision metric measures how precisely the actual output matches the expected output using purely algorithmic methods — no LLM calls required.

How It Works¶

The metric combines five sub-scores using a weighted power mean:

graph TD
    A[Actual Output] --> C[1. Exact Match]
    B[Expected Output] --> C
    A --> D[2. Character Similarity]
    B --> D
    A --> E[3. Token Precision]
    B --> E
    A --> F[4. Numeric Agreement]
    B --> F
    A --> G[5. Token Containment]
    B --> G
    C --> H[Weighted Power Mean]
    D --> H
    E --> H
    F --> H
    G --> H
    H --> I[Final Score 0.0-1.0]

Sub-Score Components¶

Default weights (from PrecisionConfig):

Component	Weight	Description
Token Containment (`contains`)	0.48	How many expected tokens appear in actual output
Character Similarity (`char_ratio`)	0.32	SequenceMatcher ratio between strings
Token Precision (`token_precision`)	0.15	Overlap coefficient of token sets
Numeric Agreement (`numeric`)	0.35	Agreement of numeric values (with tolerance)

Weights are normalized internally; an exact-match case short-circuits to 1.0.

Parameters¶

Constructor signature: AnswerPrecisionMetric(model, threshold=0.8, verbose=False, config=None).

Parameter	Type	Default	Description
`model`	`str`	required	LLM model identifier (unused for scoring but required by `MetricPattern`)
`threshold`	`float`	`0.8`	Minimum score to pass
`verbose`	`bool`	`False`	Print scoring breakdown
`config`	`PrecisionConfig`	`PrecisionConfig()`	Tuning knobs — see below

Fine-grained tuning is done via PrecisionConfig:

Field	Default	Description
`token_stopwords`	English stopwords	Tokens to drop before comparison
`numeric_tolerance_abs`	`0.0`	Absolute tolerance for numbers
`numeric_tolerance_rel`	`0.0`	Relative tolerance for numbers
`power_p`	`0.3`	Power-mean exponent
`weights`	see table above	Per-component weights

from eval_lib import AnswerPrecisionMetric
from eval_lib.metrics.answer_precision_metric.answer_precision import PrecisionConfig

metric = AnswerPrecisionMetric(
    model="gpt-4o",
    threshold=0.8,
    config=PrecisionConfig(numeric_tolerance_rel=0.05),
)

Required Fields¶

Field	Required
`input`	Yes
`actual_output`	Yes
`expected_output`	Yes

Expected output required

Unlike most metrics, Answer Precision requires expected_output since it compares actual vs. expected outputs directly.

Usage¶

from eval_lib import AnswerPrecisionMetric, EvalTestCase, evaluate
import asyncio

test_case = EvalTestCase(
    input="What is 2 + 2?",
    actual_output="The answer is 4.",
    expected_output="4"
)

metric = AnswerPrecisionMetric(model="gpt-4o", threshold=0.8)

results = asyncio.run(evaluate([test_case], [metric]))

Cost¶

0 LLM API calls — this metric is entirely algorithmic, making it the fastest and cheapest metric available.

Best Use Cases¶

Factual Q&A where answers have clear expected values
Numeric outputs (financial calculations, statistics)
Classification tasks where output should match a label
Any evaluation where you need a fast, deterministic score