This version aligns capabilities and versions across Deepchecks Multi-tenant SaaS alongside SageMaker Partner AI Apps, towards the AWS re:Invent launch.

This version includes new history field, enhancements to llm properties and improved explainability highlighting, along with more features, demos, stability and performance improvements, that are part of our 0.20.0 release.

Deepchecks LLM Evaluation 0.20.0 Release

• 💬 New “History” field
• 🏦 LLM properties bank enhancements
• 🟣 Multiple line highlighting for property explainability
• 🍿 Use case demos: Classification and Guardrails
• 📩 Data logging: partial interaction logging, steps download and upload

What’s New and Improved?

• New special field: History

  • For supplying previous historical context, such as chat history. Relevant properties will use the “History” field as additional context for checking property values.

    

• LLM properties bank enhancements

  • Added new prompts and improved prompt performance. Includes unifying the “Completeness” prompt template into one (Non-Q&A use cases have the “coverage” built-in property for uncovering issues such as a non-complete summary).

    

• Multiple line highlighting for explainability

  • Now properties such as “Grounded in Context”, “PII”, can display more than one area attributing to the highest/lowest scores, allowing efficient RCA

    

• New demos

  • Classification Demo for working with deepchecks on a classification use casue
  • Guardrails for configuring guardrails in production

• Data logging

  • An interaction can now be gradually logged, in separate parts, useful for example for production flows: Stream Upload Documentation.
  • Interaction steps can now be downloaded and uploaded via csv and SDK

This version includes expanded explainability for properties, multi-category property support, multi-label classification support, and enhancements to the documentation, along with more features, stability and performance improvements, that are part of our 0.19.0 release.

Deepchecks LLM Evaluation 0.19.0 Release

• 🌈 Highlighting of properties for explainability
• 🎡 Multi-label support for properties and classification use cases
• 🗒️ Docs Additions: Data Hierarchy and SDK Guide
• ➕ Updates to Auto-annotation flow and to Steps upload

What’s New and Improved?

• Highlighting of properties for explainability

  • Explainability highlighting for more properties: PII, Information Density, Coverage

• 

  Multi-label support for properties and classification use cases

  • Classification Demo: Movie Genre

• 

  Docs Additions

  • Hierarchy & Data Upload Format
  • SDK Quickstart

• Updates to Auto-annotation flow and to Steps upload

  • While new recalculation is in progress, previously estimated annotations will be changed to state “Pending”, and then overridden by new estimate
  • Information retrieval support going forward: only as a designated field, and not as an information retrieval “step” ⚠️

This version includes improved understanding of your version’s performance with root cause analysis, added visibility the system’s usage, along with more features, stability and performance improvements, that are part of our 0.18.0 release.

Deepchecks LLM Evaluation 0.18.0 Release

• 💡 Version Insights Enhancements
• 🔎 Score Reasoning Breakdown
• 📶 Usage Plan Visibility
• 🦸‍♀️ Improvement to PII Property
• ⚖️⚖️ Versions Page Updates
• 📈 Production Overtime View Improvements

What’s New and Improved?

• Version Insights Enhancements

  • Explainable insights with analysis, and actionable suggestion. Insights are based on property values. They Can be seen in the "Overview" screen per version, and in "Versions" screen to see application-wide insights with link to the relevant version.

    

• Score Reasoning Breakdown

  • According to Annotation Reason. Click the “Show breakdown” next to the Score on the Dashboard.

    

• Usage Plan Visibility

  • Displaying your stats and limits (Applications, Users, Processed data tokens) in the "Usage Tab" in the workspace settings

    

• Improvement to PII Property

  • Combined several mechanisms to widen detection and improve detection recall and precision on a wide set of benchmarks

• Versions Page Updates

  • Added high level data on version: insights on specific versions, and widget showing the most recent version, best performing one, etc.

    

• Production Overtime View Improvements

  • Default view now loads time ranges of most recent weeks of data

    

This version which includes a new overtime view for property values and score for production data, and exciting property related improvements, along with more features, stability and performance improvements, that are part of our 0.17.0 release.

Deepchecks LLM Evaluation 0.17.0 Release

• 📉 Overtime Production View for Monitoring
• 💬 New Property: Information Density
• 🏃🏼‍♀️‍➡️ Ability to Rerun Annotation Pipeline on Multiple Versions in Application
• 🔤 OpenAI Support for LLM Properties
• 🥑 Improvements to Relevance and Grounded in Context Properties

What’s New and Improved?

• Overtime Production View for Monitoring

  • In Production Environment, annotation scores and property scores are displayed over time

    

  • Timestamps are taken from the "started_at" field for each interaction. If no timestamp was give, current time of upload will be considered as interaction time.

• New Property: Information Density

  • Information density is a score between 0 and 1, measuring the ratio of statements that convey information (e.g. facts, suggestions), out of all statements in the output. Read more about it in the Information Density Property documentation.
  • It helps finding places where the outputs aren’t actually useful, whether if the desired information is missing (e.g. the answer is not complete), avoided, or very general without directly addressing the user’s desire.

• Rerun Annotation Pipeline on Multiple Versions

  • After uploading a new customized “Auto-Annotation YAML” in the Annotation Config screen, the annotation can now be conveniently rerun on all versions and environment in application, with the “Run Annotation Pipeline” button on the top right.

    

• OpenAI Support for LLM Properties

  • LLM Properties can now run using OpenAI (vs. Azure OpenAI). This setting is organization wide, and can be enabled by request.

• Improvements to Relevance and Grounded in Context Properties

  • Improved recall and precision on a wide set of benchmarks. Improving accuracy of Grounded in Context score also when grounding information is distributed between different documents.

This version includes improvements to the grounded in context model, explainability for grounded in context and retrieval relevancy properties, supporting GPT-4o in the backend, english translation visibility, along with more features, stability and performance improvements, that are part of our 0.16.0 release.

Deepchecks LLM Evaluation 0.16.0 Release

• 🤫 Updated Model for the Grounded in Context Property
• 🧮 Properties Explainability
• 💬 Updated LLM Reasoning View
• 🔡 Translation to English is Viewable
• 🚦 GPT-4o Supported in LLM Properties

What’s New and Improved?

• Updated Model for the Grounded in Context Property

  • Improved recall and precision on a wide set of benchmarks, enabling accurate identification of problems.

• Properties Explainability

  • Upon click, Grounded in Context displays the least grounded part of output, and similarly Retrieval Relevance marks the most relevant part of the information retrieval.

    

• Updated LLM Reasoning View

  • LLM reasoning is now viewable upon click

    

• Translation to English is Viewable

  • For applications where translation is enabled, the translated english version can be viewed
  

• GPT-4o Supported in LLM Properties

  • LLM Properties can now run using GPT-4o. This setting is organization wide, and can be enabled by request.

This version includes SDK usage improvements, monitoring flow with Datadog integration, avoided answer property updates, along with more features, stability and performance improvements, that are part of our 0.15.0 release.

Deepchecks LLM Evaluation 0.15.0 Release

• 🖥️ SDK Updates for Ease of Use
• 🤫 Improved Avoided Answer Property
• 👀 Datadog Monitoring Integration
• 🧮 Robust Tiers and Usage Tracking

What’s New and Improved?

• SDK Updates - important note for SDK use of deepchecks-llm-client>=0.15.0

  • SDK client (previously dc_client ) is now created by instantiating DeepchecksLLMClient.
  • All existing functions, such as log_batch_interactions, now require the app_name, version_name and env_type arguments to enable better control of logged data.

• Avoided Answer Property

  • New and improved algorithmic implementation for the avoided answer property.
  • Achieving an F1 score of above 0.98, on a variety of datasets (with the threshold of 0.5)

• Datadog Monitoring Integration

  • See full documentation here: Datadog Integration

    

  • Deepchecks now suppports seamless integration with Datadog, in order to easily track the metrics over time, view configurable dashboards and receive alerts. This enables Datadog users to have the relevant logs, dashboards and alerts, viewable and configurable all in one place.

• Robust Tiers and Usage Tracking

  • Monthly usage is now aggregated to allow automatic limits and billing, according to the product tier.

This version includes improvements of using the properties (improved properties screen, multiple columns in data screen, improved retrieval relevance property), output text difference highlighting, visibility for the used tokens, and more features, stability and performance improvements that are part of our 0.14.0 release.

Deepchecks LLM Evaluation 0.14.0 Release

• 🧮 New Properties Screen
• 🔢 Multiple Columns in Data Screen
• 👻 Improved Retrieval Relevance Property
• 🧑‍🤝‍🧑 Text Difference Highlighting
• 🛤️ Usage Visibility

What's New and Improved?

• New Properties Screen

  • Following your feedback, the properties experience was now improved, to enable searching and viewing all properties, no matter their type, conveniently.

  • Starred properties are the ones that are viewable in the Overview screen. Remove or add them to overview screen from the properties page itself or from the overview page.

    

  • Custom properties now support underscores in the property name.

• Multiple Columns in Data Screen

  • You can now choose multiple properties (or similarity) columns to display simultaneously in the data page, allowing to conveniently view multiple values for the interactions.

    

• Improved Retrieval Relevance

  • The “Retrieval Relevance” property is now being calculated using an improved method, leading to better detection of cases in which irrelevant information has been retrieved as part of RAG systems.

• Text Difference Highlighting

  • You can now turn “on” the toggle in the output view when comparing interactions, to highlight the differences between the outputs of the same interaction across two different versions.

    

• Usage Visibility

  • Token usage in the system is now tracked and displayed in the new “Usage” screens in the workspace settings. The token tracking mechanism will replace the existing limits on the number of interactions uploaded or on calculating LLM properties.

    

This version includes enhancements to the version comparison, additional similarity metrics: ROUGE and BLUE, expansions to the insights mechanism, and more features, stability and performance improvements that are part of our 0.13.0 release.

Deepchecks LLM Evaluation 0.13.0 Release

• 👀 Similarity additions: added ROUGE and BLEU and allow sorting by similarity
• 👭 Version comparison improvements: versions metadata, updated versions screen and comparison dialogue
• 💡 Expanded insights mechanism
• 📂 Application can now be created with SDK

What's New and Improved

• Similarity Additions

  • ROUGE and BLEU metrics are now calculated between the outputs of every two similar interactions across versions (marked by having the same user_interaction_id), in addition to the existing Deepchecks similarity.

  • In the Data screen and in the Versions screen, the Similarity column can now be used for sorting.

    

• Version Comparison Improvements

  • New design for Versions Screen, allowing to alternate between environments or expand a single version on click, to see all of its environments. Versions can now be compared across multiple properties.

    

  • Version's metadata can be added when creating a new version or editing a version. The version description is viewable upon hover in comparison screen, other fields are viewable upon opening the edit mode.

    

  • When choosing to see different interactions across versions, interactions can now be browsed to together (when scrolling and exploring different interaction sections)

• Expanded Insights Mechanism

  • Insights for weak segments detection now run on all characteristics, including: topics, custom properties, and LLM properties, in addition to the built-in properties as before.

• Create Application via SDK

  • Either as part of the init of the Deepchecks Client (see relevant section in SDK Quickstart), or with the create_application function in the SDK.

This page includes updates from our 0.12.0 Release, which includes new features, stability and performance improvements.

Deepchecks LLM Evaluation 0.12.0 Release

• 🦠New: Pentest Environment for detecting vulnerabilities that your LLM app is prone to
• 🔠 Improved Support for Non-English Use Cases
• 🗒️ Docs Additions: E2E Use case (GVHD), and new demonstrations for many evaluation features
• 🪞"Golden Set" Environment Renamed to "Evaluation" in the UI

🚧

SDK Breaking Changes

The api for determining the version and environment the SDK is set to has changed.

Previously as a method:

Python

dc_client.env_type(EnvType.PROD)
dc_client.version_name('v1')

Now, updating the class member:

Python

dc_client.env_type = EnvType.PROD
dc_client.version_name = 'v1'

What's New and Improved

• Pentest Environment
  • A dedicated environment for testing your system against known attack types. Enable it in the "Workspace Settings" to check it out.
  • Includes Pentesting data for running on your app, which should then be uploaded to Deepchecks to get an evaluation of your app's resilience to different types of attacks.
  • For more info, check out: Pentesting Your LLM-Based App
• Improved Support for Non-English Use Cases
  • Deepchecks now includes built-in support for additional languages.
  • Reach out to us to have it enabled for your organization.
• Docs Additions
  • New GVHD Use Case : Q&A Example use case to run through and get familiar with the system and its values
  • Deep explanations about features and value, such as Hard Sample Mining for Fine-Tuning can be found in the "Usage Scenarios" section.