Tracing on the Deepchecks System

Deepchecks supports uploading and displaying tracing data alongside interactions and sessions. This allows users to evaluate not only the quality of outputs, but also to track operational performance metrics — such as latency, token usage, and more. Tracing is a core part of the evaluation process, helping teams assess both the quality and efficiency of their LLM pipelines.

What Is Tracing Data?

Tracing data refers to metadata that captures how each interaction was processed. While evaluation focuses on the quality of outputs, tracing provides insight into performance and resource consumption.

Currently, Deepchecks supports the following tracing fields at the interaction level:

Latency

Users can provide the start and end time of each interaction. Deepchecks calculates and displays the total latency (duration in milliseconds).

Token Usage

Users can provide the total number of tokens used for each interaction — covering all sources, such as input, output, reasoning, or others. Deepchecks stores and displays this aggregated token count. If no token data is provided, Deepchecks automatically estimates it by parsing the interaction content.

Metadata

Users can attach any custom key:value metadata related to tracing via the SDK. This is useful for logging details such as cost, , cache status and more.

Session-Level Aggregation

Tracing data is also aggregated from the interaction level to the session level:

• Session Latency: Total latency across all interactions in the session.
• Session Token Usage: Combined total token count, provided or parsed by Deepchecks, across all interactions in the session.

❗️

Missing Tracing Data

Note: If any interaction is missing latency data, session-level latency will not be calculated and displayed.

How You Can Use Tracing Data

Once uploaded, tracing fields are accessible throughout the Deepchecks platform:

• Interaction & Session Views: Tracing data is displayed alongside other metrics for full visibility.
• Data Page: Use tracing fields to filter, search, or sort the data within a certain version of your pipeline.
• Experiment Configuration: Tracing data can be used as part of your experiment metadata, allowing you to compare versions not just by quality properties but also by runtime behavior (e.g., latency or token cost).

By leveraging tracing data within Deepchecks, teams gain visibility into both quality and efficiency — improving observability, experiment analysis, and production reliability.