Deepchecks can assist annotation workflows in several ways, by utilizing the properties and annotations in the system. These can be incorporated both when annotating with an external tool (by downloading the data, or consuming it via SDK), or within the Deepchecks UI.

Utilizing Deepchecks Automatic Annotations to Assist With Manual Annotation

Following is an example for one such flow of annotating within the Deepchecks system:

1. Upload one or several versions for which you would like to have human-verified annotations. If you have any annotated versions with similar interactions (e.g. the same golden set on a previous version), make sure you upload it as well.
2. Go to the data screen, and select the version you want to annotate.
3. Filter on the “thumbs up” to start by reviewing the “estimated as good”.

4. Clicking on an interaction will display it, allowing you to view the data from the various steps, and the values of the selected properties.

5. You can click the annotation symbol at the top to update the annotation as desired, while reviewing the various property scores and estimated annotation reasoning to enhance annotation confidence and speed.
6. In addition, you can compare the current interaction to other interactions (in different versions), enabling to annotate them simultaneously. In case that previously annotated versions where uploaded to the system, some of the estimated annotations are likely to be “due to similarity”. In that case you can see the similarity score alongside the highlighted differences in the output, and efficiently confirm the annotation.

7. You can quickly browse through the interactions (both in the comparison view or in the full view) using the arrows at the top, to continue the annotation flow.
8. Once you finished annotating all of the “recommended good”, you can do the same for “recommended bad”, and for the “unknown”.
9. This order is recommended as you will likely want to inspect similar properties and will have a grasp of the recurring problems in your application.

Note: Deepchecks can also support a similar flow for additional types of scores alongside “thumbs up” and “thumbs down” annotations (e.g. float format associated with a specific property), by treating them as “custom properties”.