📖

This Page Includes

• Deepchecks' main interaction types: 💡Q&A , 📚Summarization, 🎨Generation, 🎯Classification, ⛏️Feature Extraction
• A few of the key properties for each interaction type
• The data structure to upload for each interaction type
• 📘 Note 📘: Deepchecks is compatible with all interaction types. Selecting your specific interaction type helps you begin with optimized default settings, which you can fully customize to your needs. If you're not sure which configuration would be the best starting point, you can choose "Other", or just consult with us!

Question Answering (RAG) 💡

An interaction type whose end goal is to answer a question provided by the end user. The interaction can contain multiple intermediate steps such as embeddings-based information retrieval in the classic RAG scheme or SQL query generation to retrieve relevant information.

Q&A Key Properties

Among the various Deepchecks' properties, we found the following to be especially useful for Q&A interactions: Grounded in Context, Retrieval Relevance, Relevance, Correctness, and Completeness.

Data to upload for Q&A interactions

• Input (str) - should contain the user-provided question.
• Information Retrieval (List[str]) - retrieved information on which the answer should be based. Either documents in the case of RAG or data retrieved from the database.
• History (List[str]) - additional context relevant to the interaction, which wasn't retrieved from a knowledge base. For example: chat history.
• (Optional) Additional Steps - In case your interaction has additional intermediate steps they should be logged under the steps mechanism. See more details here.
• (Optional) Full Prompt (str) - the full prompt that was sent to the LLM.
• Output (str) - the final response that was displayed to the user.

In cases where the question is asked after a chat between the user and the application, the chat history should be included as part of the input.

See GVHD Q&A Example for a complete RAG interaction quickstart.

Summarization 📚

An interaction type designed to condense text into either a free-form summary or a specified format. Examples of such interactions are meeting-summary bots and bots tasked with summarizing a legal case or a financial report.

Summarization Key Properties

Among the various Deepchecks' properties, we found the following to be especially useful for summarization interactions: Coverage, Grounded in Context, Information Density, and Coherence.

Data to upload for Summarization interactions

• Input (str) - The text to be summarized.
• (Optional) Additional Steps - In case your interaction has additional intermediate steps they should be logged under the steps mechanism. See more details here.
• (Optional) Full Prompt (str) - the full prompt that was sent to the LLM.
• Output (str) - the generated summary.

Generation 🎨

Interactions that utilize data to perform a creative task of some sort. This is the most diverse interaction type and can include several different kinds of interactions. To name a few specific examples:

• Report Generation - receives information about a company or real estate asset, and generates a report in a specific requested format.
• Content Generation - receives details about a subject and generates a web page. For example, generate an "About Us" page on a company's website.
• Docstring Generation - receives a code snippet describing a class or a function and generates a high-quality docstring for the input code.

Generation Key Properties

Among the various Deepchecks' properties, we found the following to be especially useful for generation interactions: Grounded in Context, Relevance, and Coherence. In addition, it is likely that different interactions could significantly benefit from additional properties so it is highly recommended to check out the full catalog.

Data to upload for Generation interactions

• Input (str) - The user's input to the interaction, without additional application instruction.
• (Optional) Information Retrieval (List[str]) - retrieved information on which the output should rely.
• (Optional) History (List[str]) - additional context relevant to the interaction, which wasn't retrieved from a knowledge base. For example: chat history.
• (Optional) Additional Steps - In case your interaction has additional intermediate steps they should be logged under the steps mechanism. See more details here.
• Full Prompt (str) - the full prompt that was sent to the LLM.
• Output (str) - the interaction's output.

Classification 🎯

Includes both multi-class as well as multi-label classification interactions in which the LLM performs the classification.

Classification Key Properties

For classification interactions Deepchecks' properties, are mainly useful for segmenting and analyzing different aspects of the user input. Specifically interesting properties for this include Sentiment, Fluency, Formality, and in relevant cases also Content Type. In classification interactions, the main tools used for auto annotation are similarity-based annotation and Deepchecks Evaluator.

Data to upload for Classification interactions

• Input (str) - The input to be classified.
• (Optional) Additional Steps - In case your interaction has additional intermediate steps they should be logged under the steps mechanism. See more details here.
• (Optional) Full Prompt (str) - the full prompt that was sent to the LLM.
• Output (str) - the classification. In the case of multi-label classification should be formatted as Class 1, Class 2, ...

Feature Extraction ⛏️

Interactions where specific pieces of information are extracted from a text by an LLM. The most common example is where data is extracted from free text into a JSON file of a predefined schema.

Feature Extraction Key Properties

There are 3 key properties for this use-case:

• Extraction Groundedness - assesses whether the extracted output is faithful to the input data. A high score indicates that the output is faithful to the input and doesn't contain hallucinations..
• Extraction Coverage - assesses whether the extracted output contains all the information required by the prompt that is provided in the input. A high score indicates that a high ratio of the required information was correctly extracted.
• Structural Validity - assesses whether the extracted output fits the structure specified in the full prompt prompt. A high score indicates that the output's structure adheres to the specifications in the prompt.

Data to upload for Generation interactions

• Full Prompt (str) - The instructions for the extraction task. May contains free text, an expected structure or both.
• Input (str) - The text from which data is to be extracted.
• Output (str) - The extracted data.
• (Optional) Additional Steps - In case your interaction has additional intermediate steps they should be logged under the steps mechanism. See more details here.