Deepchecks enhances manual annotation workflows by leveraging property scores and automatic annotations. These tools can be used in external annotation platforms—by downloading the data or accessing it via the SDK—or directly within the Deepchecks UI.

Using Automatic Annotations to Guide Manual Review

Here’s an example workflow for annotating directly within Deepchecks:

Upload one or several versions for which you would like to have human-verified annotations. If you have any annotated versions with similar interactions (e.g. the same golden set on a previous version), make sure you upload it as well.
Go to the data screen, and select the version you want to annotate.
Filter on the “thumbs up” to start by reviewing the “estimated as good”.

Clicking on an interaction will display it, allowing you to view the data from the various steps, and the values of the selected properties.

You can click the annotation symbol at the top to update the annotation as desired, while reviewing the various property scores and estimated annotation reasoning to enhance annotation confidence and speed.
In addition, you can compare the current interaction to other interactions (in different versions), enabling to annotate them simultaneously. In case that previously annotated versions where uploaded to the system, some of the estimated annotations are likely to be “due to similarity”. In that case you can see the similarity score alongside the highlighted differences in the output, and efficiently confirm the annotation.

You can quickly browse through the interactions (both in the comparison view or in the full view) using the arrows at the top, to continue the annotation flow.
Once you finished annotating all of the “recommended good”, you can do the same for “recommended bad”, and for the “unknown”.
This order is recommended as you will likely want to inspect similar properties and will have a grasp of the recurring problems in your application.

Note: Deepchecks can also support a similar flow for additional types of scores alongside “thumbs up” and “thumbs down” annotations (e.g. float format associated with a specific property), by treating them as “custom properties”.