Properties

Properties are one-dimensional values that are calculated for each text sample. They can have numeric or categorical values. For example, a property could be simple text characteristics such as the number of words in the text, or more complex properties such as identifying the text language, detecting hallucination likelihood, or checking to what extent a given summary captures the original article's key points. They are built to allow automatic scalable evaluation, alongside quick root cause analysis.

What kinds of properties are there?

Built-in Properties 🏠

• Built-in properties are created and maintained by Deepchecks, and exist for every new application (they can be added or removed). They range from simple calculations to complex proprietary implementations, and some utilize LLM calls. Examples include Grounded in Context, Avoided Answer, Retrieval Relevance, Toxicity, and Sentiment. Click here for a list including many of Deepchecks' built-in properties.

Prompt Properties 🖥️

• Prompt properties utilize LLM calls for evaluations. These include predefined and customizable prompt properties that run through an LLM API, with templates provided by the Deepchecks team.

User-Value Properties ㊕

• User-Value properties are values calculated and sent by the user. They can be categorical or numerical, and can include any relevant metrics or metadata which are relevant for evaluation or RCA.

What are properties used for?

Properties measure various aspects of our LLM interactions that we may be interested in. They are used in the following ways:

• Properties are used to calculate the estimated annotations. By defining rules on the calculated properties, you can create a flow that automatically estimates the quality of each LLM interaction. For example, by default summarization interactions with low Conciseness are deemed to be bad interactions.

  

• Average values of calculated properties are shown in the Overview Screen and the Versions Screen. You can then compare property values across versions, or dive into a specific property and see interactions with extreme values, such as extremely irrelevant answers.

• Properties are shown on the Data Screen, and can be used there to sort and filter the viewed interactions. This is useful for example if you wish to see only the interactions with Toxicity > 0.5, and perhaps combine that filter with additional ones (e.g., a specific topic). In addition, property explainability can be used to quickly pinpoint the problem when inspecting a specific interaction.

Active Properties List

The property list provides a centralized view of all calculated properties within the "Properties" screen. These properties are integral to evaluations conducted during new interactions for each interaction type.

Properties not featured in this list (inactive properties) reside in the "Property Bank" and are excluded from automatic calculations, maintaining a focus on relevant data.

Each property entry includes key details: name, type (built-in, user-value, or prompt), along with a detailed description. You can also manage properties through options to remove them from the list, pin them to the overview screen, and perform additional actions.

Add Properties

To add a property to the interaction type, click "Add Properties" located at the bottom of the list. From there, the process differs based on the type of property you wish to add.

Built-in Properties:
These properties are readily available. Upon clicking, all built-in properties are displayed in the bank and can be selected for addition.

Custom Properties:
To create a custom property, select "Create Custom Property." This action opens a window where you can specify either a user-value property or a prompt property. For detailed instructions on creating a custom prompt property, refer to the prompt or user-value properties pages.

📘

Recalculate Properties

When a new property is added, it will be automatically applied to all of the new interactions sent to the system. You can use the "Recalculate" button in order to trigger the properties calculation for existing interactions. If you "recalculate" for interactions that already have values, the "recalculate" will override them. Note: for properties that utilize LLM calls, this recalculation is part of the "LLM Tokens Processed" in system usage.

Remove Properties

To remove a property from the list, simply click the remove icon next to the property.

🫡

Property removal notification

If you receive the following notification when trying to remove a property:

The property won't be removed. It means that the property is included in the YAML file that configures automatic interaction annotation, and therefore removal will be restricted by the system. To enable removal, you'd need to first Customize the Automatic Annotation Configuration to update the estimated annotations pipeline and remove that property from the YAML. Upon update, you'll be able to remove the property.