Tracing¶

Eval AI Library ships with a lightweight tracing subsystem for production monitoring of LLM calls and agent runs. It captures span data (timing, tokens, cost, input/output, tool calls, reasoning steps) and ships it asynchronously to a collector. Ten framework integrations plug into the tracer without any user glue code.

Lite Installation¶

For tracing in production without the full evaluation framework:

pip install "eval-ai-library[lite]"

The lite extra pulls in only pydantic and aiohttp, no ML dependencies.

Configuration¶

TracingConfig reads three env vars for the base install and a few opt-in knobs for tuning:

Env var	Default	Description
`TRACING_ENABLED`	`false`	Master switch. Set to `true` to enable tracing
`TRACING_URL`	`""`	HTTP endpoint of your trace receiver
`TRACING_PROJECT`	`default`	Project id used to segregate traces on the receiver
`TRACING_API_KEY`	—	Optional Bearer token for the receiver
`TRACING_SINK`	`http`	Default sink kind — `http`, `memory`, or `file`
`TRACING_SINK_PATH`	`traces.jsonl`	Path used when `TRACING_SINK=file`
`TRACING_STRICT`	`false`	When `true`, send failures raise instead of being logged (useful in CI)
`TRACING_STREAM`	`false`	When `true`, every `end_span()` immediately flushes the span as a `partial_span` — long-running sessions survive crashes

from eval_lib.tracing import TracingConfig

TracingConfig.is_enabled()  # bool
TracingConfig.get_url()     # str
TracingConfig.get_project() # str

Everything on TracingConfig is static — it just reads the environment. No constructor call needed.

Span Types¶

`SpanType`	Meaning
`LLM_CALL`	LLM API call (input messages, output, tokens, cost)
`TOOL_CALL`	Tool / function call
`AGENT_STEP`	Agent reasoning / planning step
`REASONING`	Chain-of-thought step
`RETRIEVAL`	Document / vector retrieval
`EVALUATION`	Evaluation-metric run
`CUSTOM`	Anything else

Core Tracer¶

The tracer singleton captures spans and dispatches them via the sink:

from eval_lib.tracing import tracer, SpanType

# Explicit span lifecycle
trace_id = tracer.start_trace("my-pipeline")
span = tracer.start_span("generate-answer", SpanType.LLM_CALL, input_data={"prompt": "..."})
# ... your LLM call ...
tracer.end_span(span, output="The answer is 42")
tracer.end_trace()

# Or via context manager
with tracer.trace("retrieval", SpanType.RETRIEVAL) as span:
    results = retrieve_documents(query)

Spans nest automatically via context variables — no manual parent-span plumbing needed.

Decorators¶

Automatic function instrumentation for hand-written pipelines:

from eval_lib.tracing import trace_llm, trace_tool, trace_step

@trace_llm()
async def call_openai(prompt: str) -> str:
    # Captures input, output, duration, cost as an LLM_CALL span
    ...

@trace_tool()
async def search_database(query: str) -> list:
    # Captures as a TOOL_CALL span
    ...

@trace_step()
async def process_request(request):
    # Captures as an AGENT_STEP span
    ...

Framework Integrations¶

All framework callbacks are lazy-loaded — importing them from eval_lib.tracing only requires the respective SDK when you actually use them. Install the extra you need alongside eval-ai-library[lite].

LangChain¶

from eval_lib.tracing import EvalLibCallbackHandler, callback_handler
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(callbacks=[callback_handler])  # global instance
# or: llm = ChatOpenAI(callbacks=[EvalLibCallbackHandler()])

LlamaIndex¶

from eval_lib.tracing import install_llamaindex_tracing

install_llamaindex_tracing()  # patches Settings.callback_manager globally

Lower-level: EvalLibEventHandler, EvalLibSpanHandler.

CrewAI¶

from eval_lib.tracing import CrewAITraceCollector

collector = CrewAITraceCollector()
crew.step_callback = collector

AutoGen¶

from eval_lib.tracing import AutoGenTraceHandler

handler = AutoGenTraceHandler()
# attach per AutoGen's runtime API

Haystack¶

from eval_lib.tracing import install_haystack_tracing

install_haystack_tracing()

Lower-level: EvalLibHaystackTracer.

Semantic Kernel¶

from eval_lib.tracing import install_sk_tracing

install_sk_tracing(kernel)

Claude Agent SDK¶

from eval_lib.tracing import ClaudeAgentTraceCollector

collector = ClaudeAgentTraceCollector()
# hand `collector.on_event` to the SDK's message stream

smolagents¶

from eval_lib.tracing import smolagents_step_callback

agent = CodeAgent(..., step_callbacks=[smolagents_step_callback])

phidata¶

from eval_lib.tracing import PhidataTraceCollector

collector = PhidataTraceCollector()

OpenAI Assistants¶

from eval_lib.tracing import OpenAIAssistantsTraceCollector

collector = OpenAIAssistantsTraceCollector()
# feed the assistant's run events

OpenTelemetry¶

Ship eval-lib spans through OTel (Jaeger, Tempo, DataDog, any OTel-compatible backend):

from eval_lib.tracing import EvalLibSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(EvalLibSpanExporter()))

Sinks¶

TraceSender accepts any Sink implementation. Three are built in:

`HTTPSink` — the default¶

Ships batched trace payloads to TRACING_URL over HTTP (Bearer-auth via TRACING_API_KEY).

from eval_lib.tracing import TraceSender, HTTPSink

sender = TraceSender(sink=HTTPSink(url="https://collector.example.com", api_key="..."))

`FileSink` — JSONL append¶

Useful for local debugging or air-gapped deployments.

from eval_lib.tracing import FileSink, TraceSender

sender = TraceSender(sink=FileSink(path="traces.jsonl"))

Or via env: TRACING_SINK=file TRACING_SINK_PATH=./out/traces.jsonl.

`InMemorySink` — tests¶

Buffers everything in-process; useful in unit tests.

from eval_lib.tracing import InMemorySink, TraceSender

sink = InMemorySink()
sender = TraceSender(sink=sink)
# ... run pipeline ...
assert len(sink.spans) == 3

Receiver Side¶

The receiving side lives in eval_lib.connector — trace_routes (FastAPI routes) and trace_receiver (storage + runtime-eval loop). Mount them behind your own auth / DB. See the tracing hardening spec in the repo for the pluggable-storage contract.

Trace Data Model¶

Spans are dataclasses in eval_lib.tracing.types:

from eval_lib.tracing import TraceSpan, SpanType

TraceSpan carries: span_id, trace_id, parent_span_id, span_type, name, start_time, end_time, input_data, output_data, metadata, status, plus LLM-specific fields (tokens, cost, model). All fields serialize cleanly to JSON.