Pricing¶

Eval AI Library tracks API costs for every LLM call. Pricing is resolved in two layers:

1. A hand-curated override table in eval_lib.price.model_pricing (preview SKUs and models that LiteLLM doesn't yet cover — e.g., Zhipu GLM, Qwen, DeepSeek, date-suffixed Claude ids).
2. For everything else, the library falls back to LiteLLM's built-in pricing catalogue, which is updated on every litellm release.

The snapshots below are illustrative only — refer to the provider's pricing page (or litellm.model_cost) for the current number.

OpenAI¶

Embedding Models¶

Google Gemini¶

Anthropic Claude¶

Other providers (via LiteLLM)¶

DeepSeek (v3.1/v3.2, R1), Qwen (max/plus/turbo/long, QwQ), Zhipu GLM (4-plus/air/long/flash), Mistral (large/small/codestral/pixtral/ministral), Groq (Llama 3.x), xAI/Grok and the rest are priced from the curated override table when LiteLLM doesn't ship them. Inspect the current numbers with:

from eval_lib import model_pricing
print(model_pricing["deepseek-v3.2"])

Cost Estimation¶

Per-Metric Cost¶

Example: Full RAG Evaluation¶

Evaluating 100 test cases with 4 metrics (Answer Relevancy + Faithfulness + Contextual Relevancy + Contextual Recall):

• LLM calls: 100 × (4 + 3 + 3 + 2) = 1,200 calls
• Estimated cost: ~$0.80 with gpt-4o

Cost Tracking in Results¶

When verbose=True (default), the evaluation summary includes cost information:

======================================================================
                     📋 EVALUATION SUMMARY
======================================================================

Overall Results:
  ✅ Passed: 3 / 3
  ❌ Failed: 0 / 3
  📊 Success Rate: 100.0%

Resource Usage:
  💰 Total Cost: $0.034200
  ⏱️  Total Time: 12.45s
  📈 Avg Time per Test: 4.15s

======================================================================

You can also access cost per metric programmatically:

for _, test_results in results:
    for result in test_results:
        for metric in result.metrics_data:
            print(f"{metric.name}: ${metric.evaluation_cost:.4f}")