Configure Auto Annotation Rules
How to select the right evaluation properties, thresholds and aggregate into a high quality evaluation flow.
This guide explains how to configure Deepchecks' auto-annotation for your LLM-based application. Auto-annotation evaluates and labels your interactions based on customizable criteria, helping you maintain quality standards at scale.
The recommended way to configure auto-annotation is through the visual editor in the UI - no YAML editing required. Navigate to the Interaction Types screen, select an interaction type, and click Edit YAML to open the visual pipeline editor. Every change you make in the UI is reflected in the underlying YAML and vice versa.
If you prefer to work with YAML directly - for version control, automation, or advanced use cases - the full configuration reference is available in The Configuration YAML.
This guide is organized as follows:
- Configuring Auto Annotation in the UI - the visual editor, block types, and how to use it
- Selecting the Right Properties - choosing which properties to include in your pipeline
- Threshold Detection - finding the right score cutoffs
- Auto Annotation Design - building an end-to-end annotation pipeline
- The Configuration YAML - YAML reference for advanced users
Auto-Annotation ScopeAuto-annotation is configured per interaction type and applies to all versions of an application. For example, an agent that performs both tool calling and report generation will have two distinct interaction types, each with its own properties and auto annotation pipeline.
Updated 30 days ago