Calibration¶

A CustomEvalMetric grades with an LLM judge. Before you trust its scores, you want to know how closely that judge agrees with your team's human decisions — and, where it diverges, how to fix the criteria, the examples, or the threshold. The eval_lib.calibration package does exactly that.

When to use what¶

Building blocks¶

All models are pydantic v2 and JSON round-trippable (they carry a schema_version), so a calibration set can be saved and reloaded.

1. Collect labelled cases¶

Wrap each reviewed row as a LabeledCase, then group them with the config under test into a CalibrationSet.

from eval_lib import EvalTestCase
from eval_lib.calibration import (
    CalibrationSet, CustomEvalConfig, HumanLabel, LabeledCase,
)

config = CustomEvalConfig(
    name="support_quality",
    evaluation_criteria=[
        "{{actual_output}} directly answers {{input}}",
        "{{actual_output}} stays grounded in {{retrieval_context}}",
    ],
    strategy="verdict",
    threshold=0.6,
)

cases = [
    LabeledCase(
        id="row-001",
        test_case=EvalTestCase(
            input="How do I reset my password?",
            actual_output="Click 'Forgot password' on the login screen.",
            retrieval_context=["Password reset starts from the 'Forgot password' link."],
        ),
        label=HumanLabel(verdict="pass", reason="Direct and correct."),
    ),
    # … ~10–30+ labelled rows
]

cal = CalibrationSet(config=config, cases=cases)

2. Measure agreement (no LLM)¶

These functions are pure — no LLM, no NumPy/SciPy. The positive class is fail (you care most about catching the answers your team would reject), and rows with no judge verdict are excluded.

from eval_lib.calibration import (
    agreement, inter_annotator, threshold_sweep, criterion_attribution,
)

report = agreement(cal.cases)            # needs ≥ min_n (default 20) for numeric metrics
print(report.accuracy, report.cohen_kappa, report.f1_fail)
print(report.confusion.false_fail.case_ids)   # rows the judge failed but humans passed

sweep = threshold_sweep(cal.cases, objective="kappa")
print(sweep.best.threshold, sweep.unstable)

attrib = criterion_attribution(cal.cases)      # per-criterion skip rate + culprits

• agreement(cases, min_n=20) → AgreementReport (accuracy + CI, precision/recall/F1 on fail, Cohen κ + CI, confusion with case ids, optional Spearman/MAE). Below min_n it returns status="insufficient" and withholds the numeric metrics.
• inter_annotator(cases) → InterAnnotatorReport (reviewer-vs-reviewer: percent agreement, Cohen κ for 2 raters, Fleiss κ for ≥3). Use it to check your labels are consistent before blaming the judge.
• threshold_sweep(cases, objective="kappa", min_recall_fail=None) → ThresholdReport (objective ∈ "kappa" / "accuracy" / "balanced_accuracy", or precision at a recall floor when min_recall_fail is set).
• criterion_attribution(cases) → list[CriterionStats] (per-criterion skip rate, and which criteria drove the false fails / false passes).

3. Improve a small labelled set — improve()¶

For the common 10–30-labelled-row case, improve() runs the whole loop and hands you a draft. It scores the current config ("before"), asks the AI analyst to diagnose the rows that disagree, builds up to four candidate configs, tunes each on its own scores, and keeps the best — repeating for several rounds.

from eval_lib.calibration import improve, Budget

ids = [c.id for c in cal.cases]

result = await improve(
    cal,
    learn_cases=ids,       # rows the analyst may learn from
    measure_cases=ids,     # rows candidates are compared on
    model="openai:gpt-4o-mini",
    budget=Budget(max_llm_calls=40),
    keep_threshold=True,   # default: keep YOUR threshold, fit criteria to it
    max_rounds=3,          # default
    rejudge=True,          # default: re-judge every labelled row with the current config
)

print(result.verdict)          # "improved" | "no_improvement"
print(result.proposed_config)  # the draft — improve() applies nothing
for rnd in result.rounds:
    print(rnd.round, rnd.winner, rnd.agree, "/", rnd.n)
print(result.remaining)        # case ids still disagreeing, with the judge's reason

The four candidates are: A threshold-only (no LLM; only when keep_threshold=False), B criteria edits, C B + few-shot examples, D anything else the analyst proposed. A candidate wins only with at least one more agreeing row than "before" (ties break toward fewer edits). ImproveResult also carries before, candidates (round 1), analysis, cost_usd, llm_calls, and stopped_by_budget.

• keep_threshold=True (default) scores candidates at the config's own threshold and fits the criteria to it — so you don't adopt a threshold silently tuned on old scores. Set it False to also sweep the threshold.
• rejudge=True (default) re-judges every labelled row with the current config, so labels given on an older metric version are still usable.

4. Optimize a larger set — CalibrationOptimizer¶

With more labels, split the set and validate the winner on a held-out slice that the analyst never sees.

from eval_lib.calibration import CalibrationOptimizer, InsufficientLabels

split = cal.split(holdout_ratio=0.3, seed=0)   # stable, stratified by human verdict

optimizer = CalibrationOptimizer(
    model="openai:gpt-4o-mini",
    max_candidates=6,
    min_labels=40,       # raises InsufficientLabels below this
    min_per_class=10,    # … or below this many pass/fail each
)

try:
    opt = await optimizer.optimize(cal, split, budget=Budget(max_cost_usd=2.0))
except InsufficientLabels as e:
    print("Not enough labels yet:", e)
else:
    print(opt.verdict)              # "improved" | "not_significant" | "worse"
    print(opt.diff)                 # what changed vs the current config
    print(opt.holdout_comparison)   # McNemar test on the held-out slice
    proposed = opt.proposed_config  # the draft

CalibrationSet.split(holdout_ratio=0.3, seed=0, pinned_train=None) is deterministic (stable hash of seed:id), stratified by human verdict, and forces every few-shot source_case_id (and any pinned_train ids) into the train side.

5. Adopt the result¶

Both improve() and optimize() give you a proposed_config (a CustomEvalConfig). Turn it into a live metric with to_metric(model) — the config is model-agnostic, so you pass the model only here:

metric = result.proposed_config.to_metric("openai:gpt-4o-mini")
# use `metric` in evaluate([...], [metric]) exactly like a hand-written CustomEvalMetric

To inspect or apply individual edits yourself:

from eval_lib.calibration import apply_proposals, diff_configs

new_config = apply_proposals(config, result.analysis.proposals, cases=cal.cases)
print(diff_configs(config, new_config))   # criteria added/removed/changed, params, few-shot

apply_proposals returns a new config (it never mutates the original); conflicting edits to the same criterion raise ValueError, and add_few_shot proposals need the cases= argument.

Budget and cost estimates¶

Every LLM-touching routine accepts a Budget; estimate the cost first so there are no surprises.

from eval_lib.calibration import (
    Budget, BudgetExceeded,
    estimate_evaluation, estimate_analysis, estimate_optimization,
)

est = estimate_optimization(cal, split, max_candidates=6, model="openai:gpt-4o-mini")
print(est.llm_calls, est.input_tokens, est.output_tokens, est.cost_usd)

budget = Budget(max_llm_calls=200, max_cost_usd=1.0)   # either/both; None = uncapped

• Budget(max_llm_calls=None, max_cost_usd=None) is asyncio-safe; a call that would exceed it raises BudgetExceeded, and the result's stopped_by_budget flag is set.
• estimate_evaluation / estimate_analysis / estimate_optimization → CostEstimate(llm_calls, input_tokens, output_tokens, cost_usd). cost_usd is filled only for models in the pricing catalog (string model ids); it is None for a CustomLLMClient or an unknown model.

Lower-level building blocks¶

If you want to drive the loop yourself:

• CalibrationAnalyst(model, batch_size=25, max_cases=300, contrast_ratio=0.2, concurrency=4, language="en") — await analyze(cal, cases=None, budget=None, threshold_target=None) reads every judge-vs-human disagreement, diagnoses each (map → reduce → propose), and returns an AnalysisResult (findings, proposals, rejected, warnings, …). The LLM only diagnoses and phrases; all counts are computed in code, the threshold is proposed by threshold_sweep, and every proposal is validated.
• await evaluate_config(config, cases, model, budget=None, concurrency=8) → EvaluationRun — re-scores the saved cases through the judge only (never your app), capturing per-case errors.
• compare(before, after) → Comparison — a paired McNemar test over two runs of the same cases, with fixed / broken case ids and significant.

API summary¶

from eval_lib import FewShotExample
from eval_lib.calibration import (
    # models
    CustomEvalConfig, HumanLabel, LabeledCase, CalibrationSet, Split, Proposal,
    # statistics (no LLM)
    agreement, inter_annotator, threshold_sweep, threshold_sweep_run,
    criterion_attribution,
    # candidates
    apply_proposals, diff_configs, evaluate_config, compare,
    # analyst / optimizer / small-set improve
    CalibrationAnalyst, CalibrationOptimizer, improve,
    # budget & estimates
    Budget, BudgetExceeded, InsufficientLabels,
    estimate_evaluation, estimate_analysis, estimate_optimization,
)

See also: Custom Metrics · Score Aggregation · API Reference.