LLM Providers¶

Eval AI Library routes all LLM calls through LiteLLM, which means every provider and model LiteLLM supports is available out of the box — no hand-maintained provider list, no wrappers to write. Upgrade litellm and new integrations show up automatically.

All metrics work with any provider through the same unified interface — just change the model string.

First-Class Providers¶

These providers have hand-tuned display names and dedicated documentation pages:

Any LiteLLM Provider — No Configuration Needed¶

Beyond the first-class list above, you can use any provider LiteLLM supports — AWS Bedrock, Google Vertex AI, Cohere, Replicate, Together AI, Perplexity, OpenRouter, Fireworks, Cerebras, SambaNova, Hugging Face, and more. Just use the LiteLLM model string directly:

from eval_lib import TaskSuccessRateMetric

# AWS Bedrock
metric = TaskSuccessRateMetric(model="bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0")

# Google Vertex AI
metric = TaskSuccessRateMetric(model="vertex_ai/gemini-1.5-pro")

# OpenRouter (gateway to 100+ models)
metric = TaskSuccessRateMetric(model="openrouter/meta-llama/llama-3.1-405b-instruct")

# Together AI
metric = TaskSuccessRateMetric(model="together_ai/meta-llama/Llama-3-70b-chat-hf")

# Cohere
metric = TaskSuccessRateMetric(model="cohere/command-r-plus")

# Perplexity
metric = TaskSuccessRateMetric(model="perplexity/llama-3.1-sonar-large-128k-online")

Credentials are picked up from the environment variables LiteLLM expects for each provider (AWS_ACCESS_KEY_ID, VERTEX_PROJECT, OPENROUTER_API_KEY, TOGETHERAI_API_KEY, COHERE_API_KEY, PERPLEXITYAI_API_KEY, etc.). See LiteLLM's provider docs for the full list of supported providers and environment variables.

For niche provider flags that aren't environment variables (e.g. aws_region_name, vertex_project, vertex_location), pass them via extra_kwargs when calling chat_complete() directly.

Model Specification Format¶

# Short form (OpenAI is default)
model = "gpt-4o"

# Full form with provider prefix
model = "provider:model_name"

# Examples
model = "openai:gpt-4o"
model = "anthropic:claude-3-5-sonnet-latest"
model = "google:gemini-2.0-flash"
model = "ollama:llama3"
model = "azure:gpt-4o"

Using LLMDescriptor¶

For programmatic provider selection you can pass the provider id as a plain string — LiteLLM will route it:

from eval_lib import LLMDescriptor

model = LLMDescriptor(provider="openai", model="gpt-4o")
model = LLMDescriptor(provider="anthropic", model="claude-3-5-sonnet-latest")
model = LLMDescriptor(provider="google", model="gemini-2.0-flash")

The Provider enum is kept intentionally small and only contains providers with native (non-LiteLLM) code paths: OLLAMA, MLX, ZHIPU, CUSTOM. Everything else (OpenAI, Anthropic, Google, Azure, Bedrock, Vertex AI, Cohere, Together, OpenRouter, Fireworks, Perplexity, DeepInfra, Cerebras, Databricks, Watsonx, Groq, Mistral, DeepSeek, xAI/Grok, Qwen/DashScope, …) is passed as a string and routed through LiteLLM.

Mix Providers in One Evaluation¶

You can use different providers for different metrics:

from eval_lib import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    CustomEvalMetric,
)

metrics = [
    # OpenAI for answer relevancy
    AnswerRelevancyMetric(model="gpt-4o", threshold=0.7),

    # Claude for faithfulness
    FaithfulnessMetric(model="anthropic:claude-3-5-sonnet-latest", threshold=0.7),

    # Gemini for custom evaluation
    CustomEvalMetric(
        model="google:gemini-2.0-flash",
        threshold=0.7,
        name="Quality",
        evaluation_criteria=[
            "{{actual_output}} directly answers {{input}}",
            "{{actual_output}} is clear and free of filler",
        ],
    ),
]

Direct LLM Calls¶

You can also make direct LLM calls using the library's client:

from eval_lib import chat_complete, get_embeddings

# Chat completion
response, cost = await chat_complete(
    llm="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7
)

# Embeddings
embeddings, cost = await get_embeddings(
    model="openai:text-embedding-3-small",
    texts=["Hello world", "How are you?"]
)

Per-Request Credential Overrides¶

chat_complete() accepts optional keyword arguments api_key, api_base, and extra_kwargs that are forwarded directly to the underlying LiteLLM call. They override whatever would otherwise be picked up from environment variables or from LLMDescriptor defaults — which makes them the recommended way to inject per-user credentials in multi-tenant hosts without mutating process-wide os.environ.

from eval_lib import chat_complete

# Supply credentials for a specific request without touching env vars
response, cost = await chat_complete(
    llm="openai:gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}],
    api_key=user_api_key,          # overrides OPENAI_API_KEY
    api_base="https://proxy.example.com/v1",  # point to a proxy or Azure endpoint
    extra_kwargs={"aws_region_name": "us-east-1"},  # niche provider flags
)

Precedence (highest wins):

1. Explicit api_key / api_base / extra_kwargs passed to chat_complete()
2. Values baked into LLMDescriptor (e.g. Zhipu's api_key)
3. Environment variables

Scope of support:

• The LiteLLM-backed helper (OpenAI, Anthropic, Google, Azure, DeepSeek, Qwen, Mistral, Groq, Grok, Zhipu, etc.) honors all three kwargs.
• Native helpers (Ollama, MLX) and CustomLLMClient ignore them — they configure credentials at construction time instead.
• Authentication failures raise LLMConfigurationError with the originating provider name in the message, so multi-tenant callers can surface a clean error to the right user.

Cost Tracking¶

All API calls return cost in USD when available. The evaluation engine aggregates costs across all metrics and test cases:

Total evaluation cost: $0.0342

See Pricing for model pricing details.