Documentation
DocumentationAPI ReferenceRelease Notes
DocumentationLog In
Release Notes
Start typing to search…

0.22.0 Release Notes

by Shir Chorev

This version adds support for multi-step workflows, by allowing different types of interactions within a single application. Properties and annotations now run on the Interaction Type level. This, alongside additional improvements such as to the Grounded in Context property, UI simplifications, stability improvements and performance enhancements, are part of our 0.22.0 release.

Deepchecks LLM Evaluation 0.22.0 Release

  • 🚀 Enhanced Support for Complex Applications
    • 🧩 New Interaction Types Layer
    • 🔄 SDK Updates
  • ☝️ Improved Grounded in Context Property
  • 🟣 Simplified Versions and Auto-annotation Screen

What’s New and Improved?

  • Enhanced Support for Complex Applications - Interaction Types

    • Applications now natively support multi-phase workflows.
    • Interaction types allow specifying a distinct type for each phase in the application, allowing to adapt the properties and evaluation for that logical phase. Supported predefined types include Q&A, Summarization, Generation, Classification, and Other.
    • For more details about configuring the Properties and annotation on the Interaction Type level, see Properties and Auto-Annotation YAML Configuration.

  • SDK/API Updates

    • The app_type parameter now determines the default interaction type for all interactions within an application. This provides a more intuitive setup and ensures consistent property evaluation.

      # Example usage
dc_client.create_application(APP_NAME,
                             app_type=ApplicationType.QA)

    • The new LogInteraction class introduces support for the optional interaction_type parameter, allowing you to specify the type of interaction directly when logging.
      Note:While LogInteractionType is still supported for backward compatibility, we recommend transitioning to LogInteraction as LogInteractionType will be deprecated in future versions.

      from deepchecks_llm_client.data_types import LogInteraction

single_sample = LogInteraction(
    user_interaction_id="id-1",
    input="my user input1",
    output="my model output1",
    started_at="2024-09-01T23:59:59",
    finished_at=datetime.now().astimezone(),
    annotation="Good",  # Either Good, Bad, Unknown, or None
    interaction_type="Generation"  # Optional. Defaults to the application's default type if not provided.
)

    • Interaction types can now be specified in SDK methods designed for creating or retrieving interactions. Methods for logging interactions, such as log_interaction and log_batch_interactions, now allow assigning interaction types during creation. Similarly, data retrieval methods like get_data and data_iterator support an interaction_types array, enabling filtering and retrieval based on specific interaction types. For more, see Deepchecks' SDK.

0.21.0 Release Notes

by Shir Chorev

This version aligns capabilities and versions across Deepchecks Multi-tenant SaaS alongside SageMaker Partner AI Apps, towards the AWS re:Invent launch.

0.20.0 Release Notes

by Shir Chorev

This version includes new history field, enhancements to llm properties and improved explainability highlighting, along with more features, demos, stability and performance improvements, that are part of our 0.20.0 release.

Deepchecks LLM Evaluation 0.20.0 Release

  • 💬 New “History” field
  • 🏦 LLM properties bank enhancements
  • 🟣 Multiple line highlighting for property explainability
  • 🍿 Use case demos: Classification and Guardrails
  • 📩 Data logging: partial interaction logging, steps download and upload

What’s New and Improved?

  • New special field: History

    • For supplying previous historical context, such as chat history. Relevant properties will use the “History” field as additional context for checking property values.

  • LLM properties bank enhancements

    • Added new prompts and improved prompt performance. Includes unifying the “Completeness” prompt template into one (Non-Q&A use cases have the “coverage” built-in property for uncovering issues such as a non-complete summary).

  • Multiple line highlighting for explainability

    • Now properties such as “Grounded in Context”, “PII”, can display more than one area attributing to the highest/lowest scores, allowing efficient RCA

  • New demos

  • Data logging

    • An interaction can now be gradually logged, in separate parts, useful for example for production flows: Stream Upload Documentation.
    • Interaction steps can now be downloaded and uploaded via csv and SDK

0.19.0 Release Notes

by Shir Chorev

This version includes expanded explainability for properties, multi-category property support, multi-label classification support, and enhancements to the documentation, along with more features, stability and performance improvements, that are part of our 0.19.0 release.

Deepchecks LLM Evaluation 0.19.0 Release

  • 🌈 Highlighting of properties for explainability
  • 🎡 Multi-label support for properties and classification use cases
  • 🗒️ Docs Additions: Data Hierarchy and SDK Guide
  • ➕ Updates to Auto-annotation flow and to Steps upload

What’s New and Improved?

  • Highlighting of properties for explainability

    • Explainability highlighting for more properties: PII, Information Density, Coverage

  • Multi-label support for properties and classification use cases

  • Docs Additions

  • Updates to Auto-annotation flow and to Steps upload

    • While new recalculation is in progress, previously estimated annotations will be changed to state “Pending”, and then overridden by new estimate
    • Information retrieval support going forward: only as a designated field, and not as an information retrieval “step” ⚠️

0.18.0 Release Notes

by Shir Chorev

This version includes improved understanding of your version’s performance with root cause analysis, added visibility the system’s usage, along with more features, stability and performance improvements, that are part of our 0.18.0 release.

Deepchecks LLM Evaluation 0.18.0 Release

  • 💡 Version Insights Enhancements
  • 🔎 Score Reasoning Breakdown
  • 📶 Usage Plan Visibility
  • 🦸‍♀️ Improvement to PII Property
  • ⚖️⚖️ Versions Page Updates
  • 📈 Production Overtime View Improvements

What’s New and Improved?

  • Version Insights Enhancements

    • Explainable insights with analysis, and actionable suggestion. Insights are based on property values. They Can be seen in the "Overview" screen per version, and in "Versions" screen to see application-wide insights with link to the relevant version.

  • Score Reasoning Breakdown

    • According to Annotation Reason. Click the “Show breakdown” next to the Score on the Dashboard.

  • Usage Plan Visibility

    • Displaying your stats and limits (Applications, Users, Processed data tokens) in the "Usage Tab" in the workspace settings

  • Improvement to PII Property

    • Combined several mechanisms to widen detection and improve detection recall and precision on a wide set of benchmarks

  • Versions Page Updates

    • Added high level data on version: insights on specific versions, and widget showing the most recent version, best performing one, etc.

  • Production Overtime View Improvements

    • Default view now loads time ranges of most recent weeks of data

0.17.0 Release Notes

by Shir Chorev

This version which includes a new overtime view for property values and score for production data, and exciting property related improvements, along with more features, stability and performance improvements, that are part of our 0.17.0 release.

Deepchecks LLM Evaluation 0.17.0 Release

  • 📉 Overtime Production View for Monitoring
  • 💬 New Property: Information Density
  • 🏃🏼‍♀️‍➡️ Ability to Rerun Annotation Pipeline on Multiple Versions in Application
  • 🔤 OpenAI Support for LLM Properties
  • 🥑 Improvements to Relevance and Grounded in Context Properties

What’s New and Improved?

  • Overtime Production View for Monitoring

    • In Production Environment, annotation scores and property scores are displayed over time

    • Timestamps are taken from the "started_at" field for each interaction. If no timestamp was give, current time of upload will be considered as interaction time.

  • New Property: Information Density

    • Information density is a score between 0 and 1, measuring the ratio of statements that convey information (e.g. facts, suggestions), out of all statements in the output. Read more about it in the Information Density Property documentation.
    • It helps finding places where the outputs aren’t actually useful, whether if the desired information is missing (e.g. the answer is not complete), avoided, or very general without directly addressing the user’s desire.

  • Rerun Annotation Pipeline on Multiple Versions

    • After uploading a new customized “Auto-Annotation YAML” in the Annotation Config screen, the annotation can now be conveniently rerun on all versions and environment in application, with the “Run Annotation Pipeline” button on the top right.

  • OpenAI Support for LLM Properties

    • LLM Properties can now run using OpenAI (vs. Azure OpenAI). This setting is organization wide, and can be enabled by request.

  • Improvements to Relevance and Grounded in Context Properties

    • Improved recall and precision on a wide set of benchmarks. Improving accuracy of Grounded in Context score also when grounding information is distributed between different documents.

0.16.0 Release Notes

by Shir Chorev

This version includes improvements to the grounded in context model, explainability for grounded in context and retrieval relevancy properties, supporting GPT-4o in the backend, english translation visibility, along with more features, stability and performance improvements, that are part of our 0.16.0 release.

Deepchecks LLM Evaluation 0.16.0 Release

  • 🤫 Updated Model for the Grounded in Context Property
  • 🧮 Properties Explainability
  • 💬 Updated LLM Reasoning View
  • 🔡 Translation to English is Viewable
  • 🚦 GPT-4o Supported in LLM Properties

What’s New and Improved?

  • Updated Model for the Grounded in Context Property

    • Improved recall and precision on a wide set of benchmarks, enabling accurate identification of problems.

  • Properties Explainability

    • Upon click, Grounded in Context displays the least grounded part of output, and similarly Retrieval Relevance marks the most relevant part of the information retrieval.

  • Updated LLM Reasoning View

    • LLM reasoning is now viewable upon click

  • Translation to English is Viewable

    • For applications where translation is enabled, the translated english version can be viewed

  • GPT-4o Supported in LLM Properties

    • LLM Properties can now run using GPT-4o. This setting is organization wide, and can be enabled by request.

0.15.0 Release Notes

by Shir Chorev

This version includes SDK usage improvements, monitoring flow with Datadog integration, avoided answer property updates, along with more features, stability and performance improvements, that are part of our 0.15.0 release.

Deepchecks LLM Evaluation 0.15.0 Release

  • 🖥️ SDK Updates for Ease of Use
  • 🤫 Improved Avoided Answer Property
  • 👀 Datadog Monitoring Integration
  • 🧮 Robust Tiers and Usage Tracking

What’s New and Improved?

  • SDK Updates - important note for SDK use of deepchecks-llm-client>=0.15.0

    • SDK client (previously dc_client ) is now created by instantiating DeepchecksLLMClient.
    • All existing functions, such as log_batch_interactions, now require the app_name, version_name and env_type arguments to enable better control of logged data.

  • Avoided Answer Property

    • New and improved algorithmic implementation for the avoided answer property.
    • Achieving an F1 score of above 0.98, on a variety of datasets (with the threshold of 0.5)

  • Datadog Monitoring Integration

    • See full documentation here: Datadog Integration

    • Deepchecks now suppports seamless integration with Datadog, in order to easily track the metrics over time, view configurable dashboards and receive alerts. This enables Datadog users to have the relevant logs, dashboards and alerts, viewable and configurable all in one place.

  • Robust Tiers and Usage Tracking

    • Monthly usage is now aggregated to allow automatic limits and billing, according to the product tier.

0.14.0 Release Notes

by Shir Chorev

This version includes improvements of using the properties (improved properties screen, multiple columns in data screen, improved retrieval relevance property), output text difference highlighting, visibility for the used tokens, and more features, stability and performance improvements that are part of our 0.14.0 release.

Deepchecks LLM Evaluation 0.14.0 Release

  • 🧮 New Properties Screen
  • 🔢 Multiple Columns in Data Screen
  • 👻 Improved Retrieval Relevance Property
  • 🧑‍🤝‍🧑 Text Difference Highlighting
  • 🛤️ Usage Visibility

What's New and Improved?

  • New Properties Screen

    • Following your feedback, the properties experience was now improved, to enable searching and viewing all properties, no matter their type, conveniently.

    • Starred properties are the ones that are viewable in the Overview screen. Remove or add them to overview screen from the properties page itself or from the overview page.

    • Custom properties now support underscores in the property name.

  • Multiple Columns in Data Screen

    • You can now choose multiple properties (or similarity) columns to display simultaneously in the data page, allowing to conveniently view multiple values for the interactions.

  • Improved Retrieval Relevance

    • The “Retrieval Relevance” property is now being calculated using an improved method, leading to better detection of cases in which irrelevant information has been retrieved as part of RAG systems.

  • Text Difference Highlighting

    • You can now turn “on” the toggle in the output view when comparing interactions, to highlight the differences between the outputs of the same interaction across two different versions.

  • Usage Visibility

    • Token usage in the system is now tracked and displayed in the new “Usage” screens in the workspace settings. The token tracking mechanism will replace the existing limits on the number of interactions uploaded or on calculating LLM properties.

0.13.0 Release Notes

by Shir Chorev

This version includes enhancements to the version comparison, additional similarity metrics: ROUGE and BLUE, expansions to the insights mechanism, and more features, stability and performance improvements that are part of our 0.13.0 release.

Deepchecks LLM Evaluation 0.13.0 Release

  • 👀 Similarity additions: added ROUGE and BLEU and allow sorting by similarity
  • 👭 Version comparison improvements: versions metadata, updated versions screen and comparison dialogue
  • 💡 Expanded insights mechanism
  • 📂 Application can now be created with SDK

What's New and Improved

  • Similarity Additions

    • ROUGE and BLEU metrics are now calculated between the outputs of every two similar interactions across versions (marked by having the same user_interaction_id), in addition to the existing Deepchecks similarity.

    • In the Data screen and in the Versions screen, the Similarity column can now be used for sorting.

  • Version Comparison Improvements

    • New design for Versions Screen, allowing to alternate between environments or expand a single version on click, to see all of its environments. Versions can now be compared across multiple properties.

    • Version's metadata can be added when creating a new version or editing a version. The version description is viewable upon hover in comparison screen, other fields are viewable upon opening the edit mode.

    • When choosing to see different interactions across versions, interactions can now be browsed to together (when scrolling and exploring different interaction sections)

  • Expanded Insights Mechanism

    • Insights for weak segments detection now run on all characteristics, including: topics, custom properties, and LLM properties, in addition to the built-in properties as before.

  • Create Application via SDK

    • Either as part of the init of the Deepchecks Client (see relevant section in SDK Quickstart), or with the create_application function in the SDK.