Skip to content

Language Detection

The LanguageDetectionMetric detects the language of the actual output and validates it against an expected language code.

How It Works

  1. Runs language detection on the actual output using the langdetect library
  2. Compares the detected language code against the expected language
  3. Returns 1.0 if the detected language matches, 0.0 otherwise

Parameters

Parameter Type Default Description
threshold float 1.0 Minimum score to pass
expected_language str required ISO 639-1 language code (e.g., "en", "fr", "de", "es", "ru", "zh-cn")

Required Fields

Field Required
input No
actual_output Yes
expected_output No

Dependencies

This metric requires the langdetect library:

pip install langdetect

Usage

from eval_lib.metrics import LanguageDetectionMetric
from eval_lib.test_case import EvalTestCase
import asyncio

test_case = EvalTestCase(
    input="Respond in French",
    actual_output="Bonjour, comment puis-je vous aider aujourd'hui?"
)

metric = LanguageDetectionMetric(threshold=1.0, expected_language="fr")
result = asyncio.run(metric.evaluate(test_case))
print(result.score)  # 1.0

Example Scenarios

Pass (1.0)

metric = LanguageDetectionMetric(expected_language="en")
EvalTestCase(
    input="Write in English",
    actual_output="The weather is beautiful today and the sun is shining brightly."
)
# Detected language is English

Fail (0.0)

metric = LanguageDetectionMetric(expected_language="en")
EvalTestCase(
    input="Respond in English",
    actual_output="El clima es hermoso hoy y el sol brilla intensamente."
)
# Detected language is Spanish, not English

Notes

  • Short outputs (a few words) may produce unreliable detection results. For best accuracy, ensure outputs contain at least one full sentence.
  • The langdetect library supports over 50 languages.