Properties

Properties are one-dimensional values that are calculated for each text sample. They can have numeric or categorical values. For example, a property could be simple text characteristics such as the number of words in the text, or more complex properties such as identifying the text language, detecting hallucination likelihood, or checking to what extent a given summary captures the original article's key points. They are built to allow automatic scalable evaluation, alongside quick root cause analysis.

What kinds of properties are there?

Built-in Properties 🏠

• Built-in properties are created and maintained by Deepchecks, and exist for every new application (they can be added or removed). They range from simple calculations to complex proprietary implementations, and some utilize LLM calls. Examples include Grounded in Context, Avoided Answer, Retrieval Relevance, Toxicity, and Sentiment. Click here for a list including many of Deepchecks' built-in properties.

Prompt Properties 🖥️

• Prompt properties utilize LLM calls for evaluations. These include predefined and customizable prompt properties that run through an LLM API, with templates provided by the Deepchecks team.

User-Value Properties ㊕

• User-Value properties are values calculated and sent by the user. They can be categorical or numerical, and can include any relevant metrics or metadata which are relevant for evaluation or RCA.

What are properties used for?

Properties measure various aspects of our LLM interactions that we may be interested in. They are used in the following ways:

• Properties are used to calculate the estimated annotations. By defining rules on the calculated properties, you can create a flow that automatically estimates the quality of each LLM interaction. For example, by default summarization interactions with low Conciseness are deemed to be bad interactions.

  

• Average values of calculated properties are shown in the Overview Screen and the Versions Screen. You can then compare property values across versions, or dive into a specific property and see interactions with extreme values, such as extremely irrelevant answers.

• Properties are shown on the Data Screen, and can be used there to sort and filter the viewed interactions. This is useful for example if you wish to see only the interactions with Toxicity > 0.5, and perhaps combine that filter with additional ones (e.g., a specific topic). In addition, property explainability can be used to quickly pinpoint the problem when inspecting a specific interaction.

Active Properties List

The property list provides a centralized view of all calculated properties within the "Properties" screen. These properties are integral to evaluations conducted during new interactions for each interaction type.

Properties not featured in this list (inactive properties) reside in the "Property Bank" and are excluded from automatic calculations, maintaining a focus on relevant data.

Each property entry includes key details: name, type (built-in, user-value, or prompt), along with a detailed description. You can also manage properties through options to remove them from the list, pin them to the overview screen, and perform additional actions.

Mandatory & Optional Fields

Each property relies on specific data fields to operate. Some fields are mandatory, while others are optional. If a required field is missing for a given interaction, the property value for that interaction will be marked as NA.

You can view the required and optional fields for each property in the Test & Edit Property screen.

For more details about how data is structured in Deepchecks, see Deepchecks Data Structure.

Add Properties

To add a property to the interaction type, click "Add Properties" located at the bottom of the list. From there, the process differs based on the type of property you wish to add.

Built-in Properties:
These properties are readily available. Upon clicking, all built-in properties are displayed in the bank and can be selected for addition.

Custom Properties:
To create a custom property, select "Create Custom Property." This action opens a window where you can specify either a user-value property or a prompt property. For detailed instructions on creating a custom prompt property, refer to the prompt or user-value properties pages.

📘

Recalculate Properties

When a new property is added, it will be automatically applied to all of the new interactions sent to the system. You can use the "Recalculate" button in order to trigger the properties calculation for existing interactions. If you "recalculate" for interactions that already have values, the "recalculate" will override them. Note: for properties that utilize LLM calls, this recalculation is part of the "LLM Tokens Processed" in system usage.

Edit or Copy Properties

Any property that relies on an LLM—whether it’s a built‑in one provided by Deepchecks or a custom prompt‑defined property—can be modified after creation, and this applies to both numerical and categorical types. To make changes, head to the Properties screen, open the Property Info Dialog for your target property, and click Edit & Test Property. While you're free to tweak most aspects of the property, note that the property name itself is immutable.

For Prompt Properties, everything except the name is editable: you can alter the guidelines, adjust the interaction steps (both required and optional), update the description, and even change the few‑shot example file. For built‑in LLM properties, you're limited to adding your own guidelines; Deepchecks then integrates these into its internal logic to fine‑tune the property for your specific use case. If the property is categorical, you can also modify the category list and toggle options like Allow classification to multiple categories or Allow classification to unlisted categories. At any point during editing, you can use the built‑in test feature to apply the updated property to a sample before saving your changes. Once edits are complete, you'll have the option to recalculate the property scores on existing data.

📘

Recalculating Edited Properties

The recalculation step upon editing a property ensures consistency—without it, values from older datasets won’t match those from newly processed ones, which makes any comparisons unreliable. That said, it is an optional process.

Remove Properties

To remove a property from the list, simply click the remove icon next to the property.

🫡

Property removal notification

If you receive the following notification when trying to remove a property:

The property won't be removed. It means that the property is included in the YAML file that configures automatic interaction annotation, and therefore removal will be restricted by the system. To enable removal, you'd need to first Customize the Automatic Annotation Configuration to update the estimated annotations pipeline and remove that property from the YAML. Upon update, you'll be able to remove the property.