Overview

The Sampling feature in Deepchecks allows users to upload large volumes of data—both during the evaluation and production phases—without necessarily evaluating every interaction immediately. This provides flexibility for users who seek comprehensive observability over their data while deferring the evaluation process for certain interactions.

By configuring sampling rates per environment, users can control the proportion of data subjected to evaluation, optimizing resource usage and maintaining the option to evaluate unevaluated data in the future.

How It Works

Configuring Sampling Rates

Within the "Manage Applications" screen, users can set the evaluation sampling rate for each environment (Evaluation and Production) individually. The sampling rate is a value between 0 and 1, representing the fraction of sessions to be evaluated:

Screenshot

🚧

Sampling Rates per Application

Note: The sampling rates are defined for each application separately and not on the organization level

Default Settings:

• Evaluation Environment: 1.0 (all data is evaluated)
• Production Environment: 1.0 (all data is evaluated)

Example Configuration:

• Evaluation Environment: 1.0 (all data is evaluated)
• Production Environment: 0.2 (20% of data is evaluated)

This configuration ensures that, for instance, in the production environment, only a subset of sessions (e.g., 20%) undergo evaluation, while the remaining sessions are stored as raw data with basic functionality. The data is randomly selected for evaluation based on the defined sampling rate.

Session-Level Sampling

Sampling operates at the session level, meaning that entire sessions are either included or excluded from evaluation. Individual interactions within a session are not sampled separately.

Deferred Evaluation

Unevaluated sessions are stored, can be explored and analyzed, and can also be evaluated at a later time. Users have the flexibility to evaluate individual sessions or groups of sessions as needed, enabling retrospective analysis and insights.

Pricing

• Evaluated Data: 250 DPUs per 1M uploaded tokens
• Unevaluated Data: 40 DPUs per 1M uploaded tokens

This pricing structure allows users to manage costs effectively by controlling the volume of data subjected to immediate evaluation.

Upcoming Features

A new interface for viewing unevaluated data is under development and will be available shortly. This feature will enhance data observability by providing access to unevaluated sessions directly within the Deepchecks platform. Until then, all unevaluated data is stored in our DB.

❗️

Data Retention

Note: Unevaluated data is stored securely in the Deepchecks database and remains available for exploration for up to 3 months from the time of upload. If you require access beyond this period, please reach out to our team to discuss extending the retention window.