Manual Annotations

Overview

Manual annotations let your team review and label individual interactions as Good or Bad. While Deepchecks automatically estimates annotations based on property scores and other signals, manual annotations allow human reviewers to override or confirm these judgments, providing ground-truth labels for your data.

Estimated vs. Manual Annotations

Every interaction can have two layers of annotation:

• Estimated - Automatically generated by Deepchecks using property scores, similarity models, or other evaluation methods. These appear by default for all interactions.

• Manual - Provided by a human reviewer. Is shown and used when exists.

  

Assigning Interactions for Review

Before team members annotate, organization Admins can assign specific interactions to reviewers:

1. Go to the Interactions screen

2. Select one or more interactions

3. Click the Assign button and choose a team member from the dropdown

4. The selected interactions are assigned to that person

   

You can filter interactions by assignee to see what's assigned to each reviewer, or filter for unassigned interactions. To remove an assignment, select the interactions and clear the assignee.

Annotating an Interaction

1. Open an interaction from the Interactions screen
2. Click Add (or Edit if an annotation already exists) next to the annotation field
3. Select Good or Bad
4. Optionally add a reason explaining your judgment
5. Click Save

The system records who annotated the interaction and when. The last editor of a manual annotation (even by a slight change to the reason) is considered the interaction's annotator. To remove a manual annotation from the system use the Delete option.

Annotating interactions is available for Members or higher-permission roles.

Filtering and Reviewing Annotations

The Interactions screen provides several ways to filter by annotation status:

• Annotation value - Filter by Good, Bad, Unknown, or Pending (no annotation yet)

• Annotation source - Filter by User Annotated (manual) or Estimated (automatic)

• Manual vs. Auto Diff - Filter to quickly identify where manual and automatic annotations disagree.

• Annotated By - Filter by the team member who provided the annotation

• Assignee -- Filter by who the interaction is assigned to