Creating Custom LLM-Based Properties

Deepchecks allows creating custom LLM-based properties. We'll define here a property, intended to detect answers that were avoided by the LLM, though they shouldn't have been.
LLM Properties are defined in the "Properties" page.

For your convenience, the following are the texts used in this example:

• Property Name: Falsely Avoided

• Description: Detecting avoided answers, that shouldn't have been avoided.

• Guidelines for the System Message field:

  1. Read the output. Determine if the output avoids answering the input's question.
  2. Read the information retrieval. Describe whether the information retrieval contains an answer to the input's question.
  3. If the output avoids answering and the information retrieval contains an answer, return a low score. In all other cases, return a high score.

• Selected Interactions Steps for Property: input, output, information retrieval

  

  Note

  This is a simplistic example of course.
  User-defined LLM Properties need to be tested and their prompts tuned in order to improve.
  A few popular examples are LLM properties that enable segmenting specific categories (e.g. extracting demographic data such as location), or for determining decisions such as "where is the problem within the pipeline", or "who is the relevant domain expert" for validating this sample.

LLM Properties run on all new interactions that are uploaded to system after their definition.
This why, after defining a new LLM property we'll automatically be prompted to recalculate the properties on existing samples.

We may rerun our property on all versions, by selecting them and clicking "Recalculate".

In less than a minute, we should expect to see the new property available in the Overview page (a number will appear, replacing the "No Values Available" status).
The new LLM property can be incorporated in the various flows, just like any other custom or built-in property: auto-annotation, slice-and-dice in the Data page, viewing the property's distribution in the Overview page, inspecting interactions with high and low values, etc.