Skip to content

Installation

Requirements

  • Python 3.9 or higher
  • At least one LLM provider API key (OpenAI, Anthropic, Google, etc.)

Install from PyPI

pip install eval-ai-library

Optional Extras

The default install is intentionally slim (~50 MB). Feature-specific dependencies are in extras, pick only what you need:

Extra When to use
lite Tracing only — pydantic + aiohttp, no evaluation code
datagen DatasetGenerator / DocumentLoader (langchain + pdf/docx/xlsx/ocr parsers)
local-models Local inference backends (MLX, transformers)
pii PIILeakageMetric with regex/NER backends
vectors SemanticSimilarityMetric, ReferenceMatchMetric (adds a local embeddings stack)
deterministic RegexMatchMetric, JsonSchemaMetric, LanguageDetectionMetric
langchain Only langchain-core — for integrating with an existing LangChain app
full Everything above (~3 GB) 
# Examples
pip install 'eval-ai-library[datagen]'
pip install 'eval-ai-library[datagen,pii,vectors]'
pip install 'eval-ai-library[full]'

Environment Setup

Set the API key(s) for the LLM provider(s) you plan to use:

export OPENAI_API_KEY="sk-..."

export ANTHROPIC_API_KEY="sk-ant-..."

export GOOGLE_API_KEY="AI..."

export AZURE_OPENAI_API_KEY="..."
export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com"
export AZURE_OPENAI_API_VERSION="2024-08-01-preview"

The Azure deployment is selected via the model string (e.g. azure:my-deployment), not via an env var.

# Ollama runs locally, no API key needed by default
export OLLAMA_API_BASE_URL="http://localhost:11434/v1"

Verify Installation

import eval_lib
print(eval_lib.__version__)

Optional: spaCy Model

Some metrics use spaCy for NLP processing. Download the English model:

python -m spacy download en_core_web_sm

Optional: Tesseract OCR

For OCR capabilities in data generation (extracting text from images):

brew install tesseract

sudo apt-get install tesseract-ocr

Download from Tesseract GitHub and add to PATH.

What's Next?