Deepchecks LLM Evaluation 0.45.0 Release

Version 0.45 brings a unified navigation overhaul, session-level properties, multi-turn data generation with personas, and a wave of customer-driven enhancements across SDK, self-hosted deployments, and evaluation quality.

Deepchecks LLM Evaluation 0.45.0 Release:

🧭 Navigation & Information Architecture Overhaul
🤖 Multi-Turn Data Generation with Personas
🎯 GIC New Scoring System
🕓 Manual Annotation History
🆕 Response Type Built-In Property
🧰 Tool Abuse & Error Detection in HPI
🛠️ SDK: App & Workspace Configuration
🔑 API Keys & Service Tokens Overhaul
📥 Granular Per-Call Usage CSV
🚦 Owner-Configurable LLM Usage Limits (Self-Hosted)
📖 New SDK Reference Docs
🪟 Persistent Interaction-View Layout

What's New and Improved?

🧭 Navigation & Information Architecture Overhaul

The platform's information architecture has been substantially rethought to make day-to-day work faster and clearer. Interactions, Sessions, and Storage are now unified into a single tabbed Data page, so you can move between related views without losing context. The previous Interaction Types screen has been replaced by a new App Configs screen that brings all app-level configuration into one tabbed interface. The sidebar has been reorganized with section dividers and titles, contextual docs links now live at the top of every main page, and the EVAL environment no longer carries the time-range picker.

🤖 Multi-Turn Data Generation with Personas

Synthetic data generation has been extended to multi-turn conversations. You can now configure user personas with multi-select trait dimensions that drive how synthetic users behave across turns, with refined guidelines for richer, more realistic conversations. Use it to stress-test agents and chatbots against varied user profiles before they hit production.

🎯 GIC New Scoring System

The Grounded-in-Context (GIC) property uses a new scoring system that better reflects how much of the response is actually grounded in the provided context.

🕓 Manual Annotation History

Manual annotations now keep a full history of past values. Whenever a reviewer overwrites a manual annotation, the previous value, reason, annotator, and timestamp are saved and shown in the annotation editor - so you can see how a label evolved, who changed it, and why. Old entries can be deleted individually, and the history is also returned by the interaction endpoints for programmatic access.

🆕 Response Type Built-In Property

A new built-in property, Response Type, classifies what kind of response your model produced (e.g. answer, refusal, clarification, etc.). It's available out of the box for any application, no setup required.

🧰 Tool Abuse & Error Detection in HPI

Tool Abuse and Error Detection are now part of the default High-Level Property Insights list and contribute to insight generation. Both properties were previously available but are now active by default for agentic applications.

🛠️ SDK: App & Workspace Configuration

The SDK now exposes first-class methods for reading and updating application and workspace configurations programmatically. Use it to script environment setup, sync configurations across deployments, and version-control your app behavior.

# read & update app config
config = client.get_application_config(application_id)
client.update_application_config(application_id, config)

🔑 API Keys & Service Tokens Overhaul

Service Tokens are a brand-new way to authenticate programmatic access to Deepchecks LLM. Instead of relying on personal user API keys for automation, you can now create dedicated service-token identities scoped to specific applications - ideal for CI pipelines, ingestion jobs, and backend services. Tokens are managed from a dedicated Service Tokens tab alongside the redesigned API Keys table, with admin controls for creation, editing, and revocation.

📥 Granular Per-Call Usage CSV

For customers who need detailed usage tracking, Deepchecks can now export granular per-call LLM usage as a CSV. Use it for internal billing, audit, or cost analysis across applications and properties.

🚦 Owner-Configurable LLM Usage Limits

In AWS SageMaker / self-hosted deployments, deployment owners can now configure monthly LLM usage limits per organization, with proactive notifications at 50%, 75%, 90%, and a hard stop at 100%. Use it to control costs and prevent runaway consumption across multi-tenant self-hosted setups.

📖 New SDK Reference Docs

The SDK reference documentation has been migrated from Sphinx to a new ReadMe.com-based pipeline powered by griffe. The result: a faster, cleaner, more navigable SDK reference that lives alongside the rest of the product documentation at docs.deepchecks.com.

🪟 Persistent Interaction-View Layout

The single-interaction view's divider placement is now persisted per browser. Adjust the panels to fit your workflow once and they'll stay that way across sessions.

Deepchecks LLM Evaluation 0.44.0 Release

Version 0.44 brings GPT-OSS model support, SDK enhancements, and several UX improvements across the platform.

Deepchecks LLM Evaluation 0.44.0 Release:

🧠 GPT-OSS Model Support
📖 Documentation Revamp
🪜 Steps Support in log_spans
📊 Total System Metrics
📥 Prompt Property JSON Download & Upload
💡 Suggested Feedback for Few Shots
🔒 SageMaker VPC Endpoint for Logs
🚨 Deprecated SDK Functions & Endpoints

What's New and Improved?

GPT-OSS Model Support

Deepchecks now supports GPT-OSS models for property evaluation. This allows self-hosted deployments to run LLM-based properties using open-source GPT models, removing the dependency on external LLM providers.

Documentation Revamp

The Deepchecks documentation has undergone a major revamp — restructured, rewritten, and expanded to cover the full platform in a clear and navigable way. Check it out here.

Steps Support in log_spans

The log_spans SDK function now accepts an optional steps parameter on each span. Steps allow you to attach arbitrary custom data to spans - any structured information you want to upload and later use for property calculations.

See more details here.

Total System Metrics

System metrics (input tokens, output tokens, total tokens, cost) now display total values alongside min/avg/max statistics. This gives a clearer picture of aggregate resource consumption per session.

Prompt Property JSON Download & Upload

Prompt property configurations can now be downloaded and uploaded as JSON files. This replaces the previous notebook-based export and makes it easier to version-control, share, and migrate prompt property configurations between environments.

Suggested Feedback for Few Shots

When refining LLM property evaluations with few-shot examples, the system now suggests feedback based on existing data. Suggestions are validated and aligned to help you build high-quality few-shot examples faster.

SageMaker VPC Endpoint for Logs

Self-hosted SageMaker deployments can now route logs through a VPC endpoint, keeping log traffic within your private network and avoiding public internet exposure.

Read the full documentation →

Deprecated SDK Functions & Endpoints

The include_extended_data parameter on GET /applications is deprecated and will be removed in v0.47. Use the new GET /applications/extended endpoint instead, which provides paginated access to application data with full metrics.

Deepchecks LLM Evaluation 0.43.0 Release

We're excited to announce version 0.43 of Deepchecks LLM Evaluation. This release focuses on deeper evaluation capabilities, streamlined property management, and hands-on annotation workflows. Highlights include new built-in properties for agent tool abuse and error detection, a redesigned single interaction view, manual annotation management, and expanded session-level property types.

Deepchecks LLM Evaluation 0.43.0 Release:

🤖 Know Your Agent (KYA) Feature Suite
🔧 New Properties: Tool Abuse & Error Detection
🔍 Structured Processed Data View
🔀 Simplified Span Extraction
📌 Pinned Properties & Clean Interaction View
⏸️ Pause & Ad-Hoc Property Calculations
✏️ Manual Annotation Management
💡 Few Shot Examples Management
🧵 Session-Level Prompt & User-Value Properties

What's New and Improved?

Know Your Agent (KYA) Feature Suite

Know Your Agent (KYA) is a complete flow for evaluating agentic applications - from connecting your deployed agent, through triggering it with AI-generated test datasets, to high-quality granular evaluation of every component in your agent's workflow.

• Enhanced Overview - The overview page now features span-name filters that let you differentiate between agents, tools, and LLM calls within your workflow. Select any agent to deep-dive into its sub-components - see how each tool, sub-agent, and LLM span performs with dedicated metrics and property scores. In addition, some general UX improvements were made to the Overview screen.

  

• Performance Summary & Suggested Properties - Two new insight components: a Performance Summary providing a concise AI-generated analysis of your version's health, and Suggested Properties that recommend prompt properties based on your data, including an add-property-with-one-click option. In addition, you can now generate failure mode analysis reports at any level, from an entire version down to a specific tool used by a specific agent.

  

• Connect Your Deployed Agent - Configure a deployment by providing your agent's endpoint URL, authentication tokens and custom headers. Deepchecks will use this connection to trigger your agent in the Simulation flow.

  

• Agent Simulation - Trigger your deployed agent against datasets directly from Deepchecks - including AI-generated datasets tailored to your application. Supports both single-turn and multi-turn conversations with parallel execution, automatic retries, full result tracking, and logging the run data back to Deepchecks for evaluation.

  

New Properties: Tool Abuse & Error Detection

Two new built-in properties expand evaluation coverage:

• Tool Abuse - Scores agent interactions based on tool usage efficiency, detecting patterns like repeated identical calls, ignoring error feedback, and retrying without adaptation. Available for the Agent interaction type. Read more about Tool Abuse here.
• Error Detection - Classifies whether an output is a valid response, a system/tool/API error, or empty. Uses a two-stage pipeline that first analyzes the output alone, then uses context to disambiguate borderline cases. Available for all interaction types. Additionally, the Avoidance property has been refined to focus exclusively on actual avoided answers (missing knowledge, policy restrictions, other). Error detection is now handled by the dedicated Error Detection property.

Changes to the Single Interaction View

Structured Processed Data View

The single interaction view now displays processed data fields in structured JSON format with syntax highlighting, instead of plain text. This makes it significantly easier to view and evaluate interactions where output format matters - such as structured responses, function calls, or API payloads.

Pinned Properties & Clean Interaction View

The single interaction view now separates properties into Pinned Properties (shown by default) and a collapsible Other Properties section. Pin the properties you care about most from the Properties configuration page, and they'll appear front and center when reviewing any interaction. N/A values are sorted to the end of each list, keeping actionable results at the top.

Simplified Span Extraction

Extracting a span name into its own interaction type is now more straightforward. When extracting a span, you can choose to either create a new interaction type or move it to an existing compatible one. Properties are automatically matched and remapped between source and target types, and feedback records are preserved during the transfer. Read more about this here.

Pause & Ad-Hoc Property Calculations

You can now pause any property to stop it from running automatically on new incoming data - useful for cost optimization, property development, or seasonal evaluations. Paused properties can still be run on-demand via the recalculation dialog, letting you test changes without impacting your pipeline.

Read the full Pause & Activate documentation →

Manual Annotation Management

A complete workflow for manual annotations is now available:

• Assign interactions to team members for review
• Annotate interactions as Good or Bad with optional reasoning
• Filter by annotator, assignee, or annotation status
• Track coverage via new manual annotation metrics on the overview page, including agreement rates between manual and estimated annotations

Read the full Manual Annotations documentation →

Few Shot Examples Management

The few shot system for refining LLM property evaluations has been redesigned. A new Few Shot tab in the property editor shows all few shot examples for a property in a sortable table. You can click any entry to edit its score, categories, and reasoning, or delete it.

Read the full Property Refinement documentation →

Session-Level Prompt & User-Value Properties

Session-level properties now support two new kinds beyond the built-in evaluations:

• Prompt Properties - Define custom LLM-evaluated session properties with your own guidelines. This includes a test-run interface to validate before deploying.
• User-Value Properties - Set manual or SDK-driven values on sessions for tracking custom metrics like business outcomes or user segments. Values can be set via the SDK with set_session_property_values() and are auto-created on first use. These values can also be provided or edited within the single-session UI itself.

Read the full Session-Level Properties documentation →

Deepchecks LLM Evaluation 0.42.0 Release

We're excited to announce version 0.42 of Deepchecks LLM Evaluation. This release focuses on deeper session-level insights, streamlined configuration, and enterprise-grade governance. Highlights include session-level properties that evaluate entire conversations, a visual auto-annotation editor that replaces manual YAML editing, multi-turn dataset support, and new audit and usage tracking tools for workspace administrators.

Deepchecks LLM Evaluation 0.42.0 Release:

🧵 Session-Level Properties
🎛️ Visual Auto-Annotation Editor
💬 Multi-Turn Datasets
📋 Audit Logs & Usage Export
🔔 In-App Release Notes

What's New and Improved?

Session-Level Properties

Evaluating individual interactions only tells part of the story. Session-level properties analyze entire multi-turn conversations to detect patterns that only emerge across multiple exchanges - like user frustration building over time, instruction drift, or whether all parts of a complex request were ultimately fulfilled.

Two built-in session properties are now available:

• User Satisfaction - Scores how satisfied the user appears throughout the conversation by detecting signals like repeated corrections, resignation, or genuine enthusiasm
• Intent Fulfillment - Evaluates whether the assistant addressed all user requests across the session, accounting for recovery from early mistakes and multi-step task completion

Each property reviews the full session transcript and returns a score, and reasoning with specific turn citations. You can set pass/fail thresholds and filter and sort sessions by property scores.

Session-level properties complement interaction-level properties to give you a holistic view of conversation quality - not just whether individual responses were good, but whether the overall experience succeeded.

Read the full Session-Level Properties documentation →

Visual Auto-Annotation Editor

Configuring auto-annotation rules no longer requires editing YAML files by hand. The new visual editor provides a drag-and-drop interface for building and refining your annotation pipeline directly in the UI.

What's new:

• Visual Block Builder - Create ordered blocks of conditional rules that determine whether interactions are annotated as Good, Bad, or Unknown

• Drag-and-Drop Reordering - Rearrange block priority with simple drag-and-drop, since the first matching block determines the annotation

• Built-in Distribution Insights - See histograms showing how your property values are distributed, with real-time previews of how many interactions each condition would match

  

The editor reads and writes the same YAML configuration used by the pipeline, so you can switch between the visual editor and the raw YAML at any time. Access it from Interaction Types → Choose Interaction Type → Edit YAML.

Read the full UI Auto-Annotation Configuration documentation →

Multi-Turn Datasets

Datasets now support multi-turn conversations, enabling you to build test suites that evaluate how your application handles extended, back-and-forth interactions - not just isolated single-turn exchanges.

What's new:

• Multi-Turn Dataset Type - Create datasets specifically designed for conversational testing, where each sample represents an entire conversation scenario
• Simulated User Behavior - Each sample defines a user goal along with behavioral dimensions like persistence, clarity, frustration, and directness, creating realistic and diverse conversation dynamics
• AI Generation for Multi-Turn - Describe your application and generate diverse multi-turn scenarios automatically, with AI creating varied user goals and combining behavioral dimensions for comprehensive coverage
• AI-Generated Labeling - Samples created through AI generation are automatically marked with a sparkle icon. If you manually edit an AI-generated sample, the label updates to reflect that it was edited, so you always know the origin and modification history of each sample

Audit Logs & Usage Export

Two new tools give workspace administrators better visibility into platform activity and resource consumption.

Audit Logs

Every create, update, and delete action across the platform is now automatically logged. Administrators can download audit logs as a CSV file for any date range, covering who performed each action, what was changed, and when. Sensitive data like API keys and credentials are automatically redacted. Access audit log downloads from the Workspace Settings.

Read more about audit logs in our documentation →

Granular Usage Export

A new Download Usage button in Workspace Settings → Usage lets administrators export token consumption data as a CSV. The export breaks down usage by application, version, environment, service type, and model, with separate input and output token counts for precise cost analysis. Select any date range to generate a detailed usage report.

In-App Release Notes

You'll now see a release notes popup when your Deepchecks instance is upgraded to a new version. The popup highlights what's new and links to the full changelog, so your team stays informed about new features without needing to check external documentation. You can dismiss it with a "don't show again" option, and it will only reappear when the next version is deployed.

Deepchecks LLM Evaluation 0.41.0 Release

We're excited to announce version 0.41 of Deepchecks LLM Evaluation. This release brings comprehensive cost visibility, smart property refinement, and powerful dataset generation capabilities. Highlights include automatic cost tracking across all interactions, AI-generated test data for agentic systems, and human-in-the-loop property feedback that makes evaluations more accurate over time.

Deepchecks LLM Evaluation 0.41.0 Release:

• 📊 Dataset Management
• 🤖 Agentic Dataset Generation
• 💰 Cost Tracking with Token-Level Visibility
• 🎯 Property Refinement with User Feedback

What's New and Improved?

Dataset Management

Datasets are now a first-class feature in Deepchecks, providing a structured way to create, manage, and run curated test collections for systematic LLM evaluation.

What datasets enable:

• Reproducible Testing - Run the same test suite across versions to catch regressions and measure improvements
• Controlled Evaluation - Move beyond random production sampling to intentional test scenarios
• Systematic Coverage - Ensure your application handles edge cases, error scenarios, and diverse inputs
• Benchmark Tracking - Compare performance, latency, and cost across versions with consistent test data

Core capabilities:

• Create & Organize - Build datasets with up to 500 samples, each containing input, optional reference output, and optional metadata

• Flexible Input - Add samples manually, upload CSV files, use AI generation (more on this below), or copy samples from your production data

• Sample Management - Edit samples directly in the UI, update metadata, delete unwanted entries

  

Datasets work seamlessly with both SDK and UI workflows. Create them programmatically, populate via CSV, or use the AI generation tools to build comprehensive test suites in minutes.

Read the full Dataset Management documentation →

Agentic Dataset Generation

Creating comprehensive test datasets for agentic systems is now as simple as describing what your agent does. The new Agents generation mode uses dimensional analysis to automatically create diverse, challenging scenarios that stress-test your agent across complexity levels, ambiguity, multi-step reasoning, and edge cases.

What's new:

• No Data Source Required - Generate purely from your agent's description
• Dimensional Coverage - Automatically tests simple, medium, and hard scenarios across multiple complexity axes
• Agent-Specific Challenges - Creates situations involving multi-step workflows, ambiguous instructions, and constraint conflicts

This joins existing generation methods (RAG for document-based apps and Pentest for security testing) to give you the right tool for every evaluation need.

Read the full AI Data Generation documentation →

Cost Tracking with Token-Level Visibility

Understanding LLM costs is now effortless. Deepchecks automatically tracks spending across every interaction by calculating costs based on input and output token usage.

What this gives you:

• Automatic Cost Calculation - Configure model pricing once at the organization level, and costs are computed automatically for every interaction
• Token-Level Tracking - Monitor input tokens and output tokens separately to understand where costs come from
• Aggregated Insights - View session-level and version-level cost totals to identify expensive patterns and compare cost-efficiency across versions
• Filter by Cost - Find your most expensive queries instantly to optimize prompts or switch to more efficient models

Cost tracking works seamlessly with both interactions and spans. For multi-step agentic workflows, costs from nested LLM calls automatically roll up to parent interactions, giving you accurate end-to-end spending visibility.

Simply configure your model pricing once, and every interaction automatically shows its cost. No additional integration required beyond logging the model, input_tokens, and output_tokens fields you're likely already tracking.

Read the full Cost Tracking documentation →

Property Refinement with User Feedback

Custom LLM properties now learn from your corrections. When a property's evaluation doesn't match your judgment, provide feedback explaining the correct score - and the property uses it as a training example for future evaluations.

What this enables:

• Continuous Improvement - Properties become more accurate and aligned with your quality standards over time
• Domain Adaptation - Teach properties your industry terminology, edge cases, and subjective quality thresholds
• No Code Required - Refine evaluations through simple UI feedback, no prompt engineering needed

How it works:

For any custom LLM property evaluation, click the feedback icon and provide:

1. Your corrected score (1-5 stars for numerical properties or category selection for categorical properties)
2. Reasoning explaining why your score is more accurate

Deepchecks incorporates your feedback as in-context learning examples, so future evaluations reference your corrections when judging similar cases. Currently, the property-enhancing feedback is available for prompt properties only.

This transforms properties from static evaluation rules into adaptive quality models that evolve with your needs.

Read the full Property Refinement documentation →

Deepchecks LLM Evaluation 0.40.0 Release

We’re excited to announce version 0.40 of Deepchecks LLM Evaluation. This release strengthens self-hosted deployments, improves how agentic systems are evaluated end-to-end, and continues to simplify platform governance. Highlights include a new self-hosted deployment guide, children-based annotation aggregation for agentic workflows and centralized model management for self-hosted environments.

Deepchecks LLM Evaluation 0.40.0 Release:

🏗️ Self-Hosted Deployment Guide
🧠 Model Management for Self-Hosted Deployments
🧩 Children Annotation Aggregation for Agentic Workflows
🔐 RBAC Safeguards for Owners
⚠️ Tool Use Interaction Deprecation

What’s New and Improved?

Self-Hosted Deployment Guide

Deepchecks now includes a dedicated deployment guide for Self-Hosted Enterprise, making it easier than ever to run Deepchecks entirely in your own infrastructure.

What this gives you:

• Deploy Anywhere - Run Deepchecks inside your own environment with full control over networking, security, and data boundaries
• Production-Ready by Design - Built for Kubernetes to support reliability, scalability, and real-world workloads
• Clear Deployment Path- A structured walkthrough of the required components and how they fit together

This guide focuses on helping teams understand what’s required and why, without forcing them to become Deepchecks experts on day one. Whether you’re deploying on AWS or adapting to another environment, the core idea is simple: Deepchecks is designed to fit cleanly into your existing infrastructure.

Model Management for Self-Hosted Deployments

Self-hosted Deepchecks deployments now include first-class model management.

Unlike SaaS deployments, self-hosted environments don’t rely on Deepchecks-managed models. Instead, you explicitly configure the models you own and operate - and Deepchecks makes them available everywhere they’re needed.

What this enables:

• Centralized Configuration - Manage all models in one place at the organization level
• Broad Provider Support - OpenAI, Azure OpenAI, AWS Bedrock, and self-hosted endpoints via LiteLLM
• Immediate Availability - Once added, models appear instantly across evaluations, applications, and preferences

Key details:

• Models are validated and connectivity-tested before being saved
• No models are configured by default - you stay fully in control

This ensures self-hosted users get the same smooth evaluation experience, without compromising ownership or security.

Children Annotation Aggregation (Agentic Evaluation)

Evaluating agentic systems requires more than looking at a single interaction in isolation. With Children Annotation Aggregation, parent interactions are now automatically annotated based on the quality of their child interactions.

Why this matters:

In agentic workflows, structure is hierarchical:

• An Agent may invoke multiple tools and LLM calls
• A Chain coordinates a sequence of steps
• A Root represents the full end-to-end execution

A parent interaction might look fine on its own - but if its children fail, hallucinate, or misuse tools, the overall outcome isn’t truly successful.

What’s new:

• Automatic Upward Propagation - Parent interactions inherit annotations based on their children
• Configurable Rules - Define thresholds (e.g. “mark bad if any child is bad” or “if >50% fail”)
• Type Filtering - Apply aggregation only to specific child interaction types (LLM, tool, etc.)

The result is a more honest, system-level view of agentic behavior - one that reflects what actually happened beneath the surface.

RBAC Safeguards for Owners

To protect organizational stability, we’ve added an important safeguard to role management.

What’s changed:

• Every organization must always have at least one Owner
• An Owner can't downgrade their own role unless another Owner already exists

This prevents accidental lockouts and ensures there’s always someone with the permissions required to manage configuration, users, and critical settings.

Tool Use Interaction Deprecation

The legacy Tool Use interaction type has been deprecated.

What to know:

• Tool Use is no longer a standalone interaction type
• All agentic workflows now use the unified agentic interaction model: Root, Agent, Chain, Tool, and LLM

This change simplifies the interaction model and better reflects how modern agentic systems actually operate, while enabling more consistent evaluation and aggregation across hierarchies.

Deepchecks LLM Evaluation 0.39.0 Release

We're excited to announce version 0.39 of Deepchecks LLM Evaluation - introducing per-application access control, enhanced prompt properties with descendant data access, Google ADK integration, and CloudWatch metrics for SageMaker deployments. This release strengthens access control granularity, expands framework support, and provides deeper insights into complex agentic workflows.

Deepchecks LLM Evaluation 0.39.0 Release:

🔐 Per-Application Access Control (New RBAC Tier)
🌳 Prompt Properties: Access Descendant Data
🤖 Google ADK Integration
💡 AI Assistant for Documentation
📊 CloudWatch Integration for SageMaker Deployments
⚠️ Deprecations & API Changes

What's New and Improved?

Per-Application Access Control (New RBAC Tier)

Deepchecks now supports a second tier of access control that enables fine-grained application-level permissions. When enabled, users can be granted access to specific applications rather than having automatic access to all applications in the organization.

What this means:

• Granular Access Control - Admins and Owners can specify which applications each user can access when inviting users or updating permissions
• Enhanced Security - Users only see and can access applications they've been explicitly granted access to

This feature is particularly valuable for organizations with multiple teams or applications that require stricter data access boundaries.

To learn more about Deepchecks RBAC and the new per-application access tier, click here.

Prompt Properties: Access Descendant Data

Prompt properties can now access data from descendant (child) spans in hierarchical workflows, enabling more comprehensive evaluation of complex agentic use-cases.

What's new:

• Descendant Scopes - Add scopes to your prompt properties to access data from child spans at different levels
• Granular Control - Specify depth levels, filter by interaction types, and select which data fields to include from descendant spans
• Flexible Configuration - Each scope can be configured independently, allowing you to gather different types of information from different parts of the hierarchy

Value:

This enhancement enables you to create highly tailored properties for agentic workflows. For example, you can evaluate whether all tool calls in an agentic interaction were appropriate by accessing all descendant Tool spans, or evaluate the quality of responses from an agent's direct LLM children by looking one level down at LLM spans only.

The Data component provides full flexibility over which data fields are sent to the LLM, ensuring token efficiency by only including what's needed for evaluation.

For detailed information on configuring descendant data access, click here.

Google ADK Integration

Deepchecks now integrates seamlessly with Google ADK, enabling automatic trace collection and evaluation for Google ADK workflows.

What you get:

• Automatic Instrumentation - Uses OTEL + OpenInference to automatically instrument Google ADK interactions
• Rich Traces - Captures LLM calls, tool invocations, and agent-level spans within the graph
• Simple Setup - Register with Deepchecks through a single register_dc_exporter call
• Full Evaluation - Benefit from Deepchecks' observability, evaluation, and monitoring capabilities

Google ADK joins the popular frameworks we're already integrated with - CrewAI and LangGraph - expanding Deepchecks' support for the most widely-used agentic frameworks.

For installation and setup instructions, click here.

AI Assistant for Documentation

You can now get instant help directly inside our documentation with the new AI Assistant.

Just click “Ask AI” next to the search field on the documentation site and ask anything you need - from feature explanations and setup guides to “how do I…” questions and best practices.

Why it’s useful:

• No more digging - Ask questions in plain language instead of searching through pages
• Context-aware answers - Get relevant guidance across features, guides, and workflows
• Faster onboarding & troubleshooting - Find what you need in seconds

Whether you’re exploring new capabilities or looking for a specific how-to, the AI Assistant helps you get answers instantly, right where you need them.

CloudWatch Integration for SageMaker Deployments

SageMaker customers can now send Deepchecks monitoring data and LLM evaluation metrics directly to AWS CloudWatch, consolidating model-monitoring data in one place.

What this enables:

• Unified Monitoring - View all your Deepchecks metrics alongside your existing AWS dashboards and alarms
• Quick Setup - Start getting metrics in CloudWatch within seconds after configuration
• Seamless Integration - Use your existing AWS monitoring infrastructure without additional tooling

How it works:

1. Add the required IAM permissions to your SageMaker execution role
2. Enable CloudWatch metrics in Workspace Settings (Owner role required)
3. Metrics automatically flow to CloudWatch under the DeepChecksLLM namespace

This integration allows you to leverage your existing AWS monitoring workflows while benefiting from Deepchecks' comprehensive LLM evaluation capabilities.

For setup instructions and IAM policy details, click here.

Deprecations & API Changes (Public Endpoints)

As part of ongoing improvements to our API surface, we’re beginning the deprecation process for several public API endpoints. These endpoints will be removed after three upcoming milestones, following the same policy used for SDK deprecations.

Endpoints being deprecated:

• GET /reference/listappversions
• POST /reference/createappversion
• POST /reference/publiccreateinteractions

SDK upgrade strongly recommended The newer SDK versions include improvements aligned with the new endpoints and permissions model, and provide clearer, more helpful messaging.

Improved permission handling With the updated access control model, API responses now return 403 in cases where:

• the application does not exist, or
• the user does not have permission to access the application. The latest SDK improves this experience by explicitly surfacing “application not found” when applicable (the most common case).

We’re excited to announce version 0.38 of Deepchecks LLM Evaluation - introducing framework-agnostic data ingestion for agentic workflows, a more expressive Avoidance evaluation property, and a new Metric Viewer role. This release expands who can use Deepchecks, improves failure signal quality, and strengthens access control and platform clarity.

Deepchecks LLM Evaluation 0.38.0 Release:

• 🧩 Framework-Agnostic Agentic Data Ingestion
• 🚫 Avoided Answer → Avoidance (Enhanced Property)
• 🔐 New RBAC Role: Metric Viewer
• 🤖 New Models Available for LLM-Based Features
• ⚠️ SDK Deprecation Notice: send_spans()

What's New and Improved?

Framework-Agnostic Agentic Data Ingestion

Deepchecks now supports uploading agentic and complex workflow data via the SDK, without relying on automatic tracing from a supported framework. This enables full observability and evaluation for teams using custom frameworks, in-house orchestration layers, or unsupported agent runtimes.

You can manually structure and send sessions, traces and spans to Deepchecks while still benefiting from the full evaluation, observability, and root-cause analysis capabilities.

For a step-by-step guide, click here.

Avoided Answer → Avoidance (Enhanced Property)

The existing Avoided Answer property has been upgraded to Avoidance, providing richer and more actionable signals.

What changed:

• Previously: a binary (0/1) score indicating whether an answer was avoided.
• Now: a categorical property that distinguishes between:
  • valid — the input was not avoided
  • Specific avoidance modes (e.g. policy-based, lack of knowledge, safety constraints, and more)

This enables clearer diagnosis of why an answer was avoided and supports more meaningful aggregation and analysis across versions.

📌 Deprecation note: The legacy Avoided Answer property is deprecated but will continue to function for existing applications.

For full property definitions and migration details, click here.

New RBAC Role: Metric Viewer

We’ve added a new role to Deepchecks Role-Based Access Control: Metric Viewer.

This role is designed for stakeholders who need high-level insights without access to raw data.

Metric Viewer capabilities:

• Read-only access to aggregated metrics and evaluation results
• Access limited to the version level
• ❌ No access (via UI or SDK) to raw spans and traces
• ❌ No write permissions

This complements the existing Viewer role by enabling stricter data-access boundaries for security-sensitive environments.

To learn more about Deepchecks RBAC roles, click here.

New Models Available for LLM-Based Features

The following models are now supported:

• GPT-5.1
• Amazon Nova 2 Lite
• Amazon Nova Pro

These models can be selected for evaluation, analysis, and automation features across the platform.

SDK Deprecation Notice: send_spans()

The SDK function send_spans() has been renamed to log_spans_file() to better reflect its behavior and usage.

📌 Deprecation notice: send_spans() is now deprecated and will remain supported for the next few releases. We recommend migrating to log_spans_file() to ensure forward compatibility.

Updated SDK documentation and examples reflect the new function name.

We’re excited to announce version 0.37 of Deepchecks LLM Evaluation — featuring enhanced Agent Execution Flow Graphs, flexible span-to-interaction mapping, comprehensive version-level failure mode analysis, CloudWatch integration, and improved hierarchical views. This release helps users navigate complex agentic workflows, consolidate failure insights, and monitor metrics with even greater clarity and control.

Deepchecks LLM Evaluation 0.37.0 Release:

• 🕸️ Enhanced Agent Execution Flow Graph
• 🔄 Map Spans to Custom Interaction Types
• 📊 Version-Level Failure Mode Summary
• ☁️ CloudWatch Integration for Metrics
• 📂 Collapsible Trees in Hierarchical Views

What's New and Improved?

Enhanced Agent Execution Flow Graph

The Agent Execution Graph now offers more interactivity and insights:

• Clicking a node filters the Interactions screen to that specific node.
• Hovering over nodes or edges shows metadata across all filtered runs.
• Node and edge styles indicate consistency: solid lines appear in all filtered runs, dashed lines if only in some.

Map Spans to Custom Interaction Types

You can now assign specific span names to custom interaction types, overriding default mappings. This allows spans with the same kind to have distinct properties or auto-annotation rules, for example, mapping a “Reader” span and a “Writer” span to separate interaction types. For more details, click here.

Version-Level Failure Mode Summary

In addition to property-level insights, Deepchecks now generates a **version-level failure mode analysis report. **It aggregates failures across all interaction types and selected properties, providing a consolidated view of dominant issues in the system. Ideal for spotting cross-cutting problems, prioritizing improvements, and detecting regressions. Access this from the Version page via “Analyze version failures.” For more information, click here.

AWS CloudWatch Integration

Deepchecks can now send monitoring data and LLM evaluation metrics directly to AWS CloudWatch. SageMaker users benefit immediately, with metrics appearing in CloudWatch dashboards and alarms without any additional setup. For more details click here.

Collapsible Trees in Hierarchical Views Hierarchical use cases now include collapse buttons, making it easier to navigate and focus on relevant branches of your workflows.

We’re excited to announce version 0.36 of Deepchecks LLM Evaluation - featuring a powerful new Agent Graph visualization, enhanced filtering and analysis capabilities, and deeper AWS SageMaker configurability. This release helps users better understand complex agentic executions, gain sharper analytical control, and manage configurations with greater transparency.

Deepchecks LLM Evaluation 0.36.0 Release:

• 🕸️ New Agent Execution Graph
• 🎯 Advanced Filtering by Span Attributes
• ☁️ Dedicated SageMaker Owner Panel
• ⚡ New Processing Status: Partial

What's New and Improved?

Agent Execution Graph

Visualize how your agentic workflows actually execute - including branches, loops, and transitions - directly within Deepchecks.
The new Agent Execution Graph provides a dynamic graph-style view of your pipeline runs, built automatically from your existing span metadata (no instrumentation changes required). It’s highly useful for agentic frameworks like LangGraph and CrewAI, helping you understand real runtime behavior at a glance.

You can find the Agent Execution Graph on the Overview page, within the Sessions tab, whenever you’re viewing an agentic use case that includes relevant span data.

Advanced Filtering by Span Attributes

You can now apply granular filters based on span-level attributes, such as span name, metadata fields, and more, directly in the Overview screen.
This enhancement gives analysts more precise control over their investigations, allowing them to isolate behaviors, patterns, or anomalies tied to specific spans or framework metadata.

Dedicated SageMaker Owner Panel

In Deepchecks in SageMaker, the user designated as the Owner now has access to a dedicated Owner Panel with advanced permissions to configure organization- and application-level settings directly through the UI.
This empowers teams to fine-tune Deepchecks behavior across environments - without code changes or redeployment.

Using the Owner Panel on SageMaker →

New Processing Status: Partial

We’ve added a third processing status: Partial.
This new state appears when an interaction or session is stuck or failed during a key processing phase, helping users easily distinguish between ongoing runs and those that will not progress further without further user-action.
This improvement brings more transparency and reliability, especially in non-SaaS or self-managed deployments.