Skip to content

Test Case Generation

Eval AI Library includes a powerful test case generator that creates evaluation datasets from your documents. It supports 15+ document formats including PDF, DOCX, CSV, JSON, HTML, and images with OCR.

Installation

Dataset generation has optional dependencies (document loaders, embeddings). Install the extra:

pip install 'eval-ai-library[datagen]'

Supported Formats

Category Formats
Text .txt, .md, .rtf, .xml, .json, .yaml, .html
Office .pdf, .docx, .docm, .xlsx, .pptx
Data .csv, .tsv
Images .png, .jpg, .jpeg (with OCR via Tesseract)

Quick Start

DatasetGenerator exposes two async entry points: generate_from_documents() and generate_from_scratch(). Both return a list[dict] — plain JSON-serializable rows which can be fed into EvalTestCase at will.

import asyncio
from eval_lib import DatasetGenerator, EvalTestCase

generator = DatasetGenerator(
    model="gpt-4o",
    input_format="question",
    expected_output_format="answer",
    agent_description="You are a helpful assistant that answers questions about machine learning.",
    test_types=["factual", "reasoning"],
    max_rows=20,
    language="en",
)

rows = asyncio.run(
    generator.generate_from_documents(["./knowledge_base.pdf"])
)

for row in rows:
    print(f"Q: {row['input']}")
    print(f"A: {row['expected_output']}")
    print("---")

# Convert to EvalTestCase when you're ready to evaluate:
test_cases = [EvalTestCase(**row) for row in rows]

Parameters

Parameter Type Default Description
model str required LLM for generation
input_format str required How inputs are formatted (e.g., "question")
expected_output_format str required Expected answer format (e.g., "answer")
agent_description str required System role/context for generation
test_types list[str] required Types of test cases to generate
question_length str "mixed" "short", "medium", "long", "mixed"
question_openness str "mixed" "open", "closed", "mixed"
chunk_size int 1024 Document chunking size (characters)
chunk_overlap int 100 Overlap between chunks
temperature float 0.3 Generation temperature
max_rows int 10 Number of test cases to generate
trap_density float 0.1 Proportion of trap/adversarial questions
language str "en" Language for generated test cases
max_chunks int 30 Maximum chunks to feed the LLM per document
relevance_margin float 1.5 Threshold for context relevance
embedding_model str "openai:text-embedding-3-small" Model for semantic similarity
verbose bool False Print progress logs

Generate from Scratch (no documents)

When you don't have source documents, generate a synthetic dataset from the agent description alone:

rows = asyncio.run(generator.generate_from_scratch())

Document Loading

Internally the generator uses two helper functions from eval_lib.datagenerator.document_loader:

from eval_lib.datagenerator.document_loader import load_documents, chunk_documents

docs = load_documents(["./data/report.pdf", "./data/faq.csv"])
chunks = chunk_documents(docs, chunk_size=1024, chunk_overlap=100)
  • load_documents(file_paths) — takes a list of paths and returns the parsed text of each file.
  • chunk_documents(docs, chunk_size=1024, chunk_overlap=100) — splits docs with character-based overlap.

Advanced Examples

Generate from Multiple Sources

rows = asyncio.run(
    generator.generate_from_documents([
        "./docs/api_reference.pdf",
        "./docs/user_guide.md",
        "./data/faq.csv",
    ])
)

Customize Question Types

# Short, factual questions
generator = DatasetGenerator(
    model="gpt-4o",
    input_format="question",
    expected_output_format="brief factual answer",
    agent_description="FAQ assistant",
    test_types=["factual"],
    question_length="short",
    question_openness="closed",
    max_rows=30,
)

# Open-ended, detailed questions
generator = DatasetGenerator(
    model="gpt-4o",
    input_format="question",
    expected_output_format="comprehensive answer with examples",
    agent_description="Technical tutor",
    test_types=["reasoning"],
    question_length="long",
    question_openness="open",
    max_rows=15,
)

With Trap Questions

Trap questions test the AI's ability to say "I don't know" when the answer isn't in the context:

generator = DatasetGenerator(
    model="gpt-4o",
    input_format="question",
    expected_output_format="answer",
    agent_description="RAG assistant",
    test_types=["factual", "trap"],
    trap_density=0.2,  # 20% of questions will be traps
    max_rows=50,
)

Multilingual Generation

# Russian
generator = DatasetGenerator(
    model="gpt-4o",
    input_format="вопрос",
    expected_output_format="подробный ответ",
    agent_description="Ассистент по ML",
    test_types=["factual"],
    language="ru",
    max_rows=20,
)

Use Generated Test Cases

import asyncio
from eval_lib import evaluate, EvalTestCase, AnswerRelevancyMetric, FaithfulnessMetric

rows = asyncio.run(generator.generate_from_documents(["./knowledge_base.pdf"]))
test_cases = [EvalTestCase(**row) for row in rows]

metrics = [
    AnswerRelevancyMetric(model="gpt-4o", threshold=0.7),
    FaithfulnessMetric(model="gpt-4o", threshold=0.7),
]

results = asyncio.run(evaluate(test_cases, metrics))

OCR for Images

For image-based documents, ensure Tesseract is installed:

rows = asyncio.run(generator.generate_from_documents(["./scanned_document.png"]))

See Installation for Tesseract setup instructions.