Session-Level Properties

Overview

Session-level properties evaluate the quality of entire session rather than individual interactions. While interaction-level properties score each span independently, session-level properties analyze the full conversation transcript to assess qualities that only emerge across multiple turns - such as whether the user left satisfied or whether all their requests were ultimately addressed.

Why Session-Level Properties?

Some quality signals are invisible at the span level. Consider a session where the assistant gives a wrong answer in turn 2 but corrects itself in turn 4 after the user pushes back. Each individual interaction might score reasonably well, but only by looking at the full session can you detect patterns like:

• User frustration building over time - repeated corrections, resignation, or the user giving up entirely
• Instruction drift - the assistant following a persistent instruction initially but gradually deviating
• Recovery from errors - early mistakes that get resolved in later turns
• Cumulative fulfillment - whether all parts of a complex, multi-turn request were eventually addressed

Session-level properties are especially valuable for long, complex sessions where individual turn quality doesn't tell the full story.

Built-In Session Properties

Deepchecks provides a few built-in session-level properties. These properties produce a numeric score from 1 to 5 and are evaluated by an LLM that reviews the session's data engineered by Deepchecks:

User Satisfaction

Measures how satisfied the user appears based on what they express throughout the conversation. This property looks for explicit satisfaction and dissatisfaction signals - not whether the answer was objectively correct, but whether the user seemed happy with the experience.

What it detects:

Note: Requires a minimum of 2 turns to evaluate. Sessions with a single turn will not receive a score.

Intent Fulfillment

Evaluates how well the assistant addressed the user's requests across the entire session. This property tracks every explicit request - including persistent instructions like "always respond in bullet points" - and checks whether each was addressed at some point during the conversation.

Key evaluation principles:

• Recovery counts - If the assistant fails initially but corrects itself later, the request is considered addressed
• Clarification is positive - Asking for clarification followed by a genuine attempt counts as addressing the request
• Addressing vs. perfection - A genuine attempt to help counts, even if the answer isn't perfect. What matters is that the assistant engaged with the request rather than ignoring it
• Critical failures cap the score - Ignoring a request 3+ times, responding to a completely wrong topic, or ignoring parts of a multi-part question cap the score at 2 or below

Adding Session Properties to Your Application

Session-level properties are added per application from the application's properties configuration.

Via the UI

1. Navigate to your application's Properties page

2. Open the Session Properties tab

3. Click Add Property

4. Select and add property from the available list

5. After adding a session-level property you can recalculate in on previous versions. This can also be done on-demand at any time.

   

Viewing Results

Session property scores appear in the Sessions property tab within in each span in the session. For each property you can view it's score and reasoning.

You can also use session session property scores to filter and sort the Sessions screen, helping you quickly find problematic sessions that need attention.