Toxicity Detection¶

The Toxicity Detection metric evaluates AI output for toxic content including mockery, hate speech, personal attacks, threats, and harmful language. It helps ensure that AI-generated content is safe, respectful, and appropriate for its intended audience.

Toxicity in AI outputs can range from overtly harmful (hate speech, threats) to subtly inappropriate (sarcasm, condescension, passive-aggressive tone). This metric catches both extremes and provides a continuous score reflecting the severity of detected toxicity.

How It Works¶

Uses few-shot LLM evaluation with specific toxicity detection criteria. The judge model analyzes the output for various types of toxic content: direct insults, hate speech targeting protected groups, threats of violence, mockery, sarcasm used to demean, and other forms of harmful language. The score reflects how free the output is from toxic content (1.0 = completely non-toxic, 0.0 = highly toxic).

Parameters¶

Required Fields¶

Usage¶

from eval_lib import ToxicityMetric, EvalTestCase, evaluate
import asyncio

test_case = EvalTestCase(
    input="What do you think about the new policy?",
    actual_output="The new policy has both strengths and weaknesses. The increased funding for education is positive, but the implementation timeline could be improved."
)

metric = ToxicityMetric(model="gpt-4o", threshold=0.7)
results = asyncio.run(evaluate([test_case], [metric]))

Scoring¶

When to Use¶

• Customer support bots — ensure the AI never responds with frustration, sarcasm, or hostility
• Social media moderation — evaluate AI-generated content and responses for appropriateness
• Content generation — check that generated text maintains a professional, respectful tone
• Any public-facing AI — as a baseline safety check before deployment

Cost¶

1 LLM API call per evaluation.