0.23.0 Release Notes

This version introduces the concept of Sessions, enabling better organization and analysis of interactions across complex workflows such as agents. This capability is now fully integrated across the platform, including SDK support for managing and interacting with session-level data. The Sessions concept, along with additional improvements, stability updates, and performance enhancements, is part of our 0.23.0 release.

Deepchecks LLM Evaluation 0.23.0 Release

• 🧮 New Sessions Layer, and SDK Enhancements Supporting it
• 🔡 Data Screen Content Search
• ⛏️ Feature Extraction Interaction Type

What’s New and Improved?

• New Sessions Layer for evaluating and viewing multi-phase and agentic workflows

  • Sessions introduce a new hierarchy for organizing interactions, allowing users to logically group related activities, such as conversations or tasks split into multiple steps.

  • When opening an interaction on the data screen, you can see all interactions associated with the same session id

  • More info about SDK adaptations below

    

• Data Screen Content Search

  • Interactions can now be searched for in Data screen based on interaction content and not only IDs.

  • This is selectable using the search filters in the Data screen.

    

• Feature Extraction Interaction Type

  • Feature Extraction is an interaction type dedicated to cases where information is extracted from a text into a predefined format (e.g. a JSON schema). This interaction type also presents three new properties that excel in evaluating an LLM's performance in an extraction task.

    

Session SDK/API Enhancements

• Session Support in LogInteraction

  • Introduced the optional session_id parameter in the LogInteraction class, enabling developers to assign custom session identifiers to group related interactions.

  • If session_id is omitted, the system generates a unique session ID automatically.

    Python

    from deepchecks_llm_client.data_types import LogInteraction
    from datetime import datetime
    
    single_sample = LogInteraction(
        user_interaction_id="id-1",
        input="my user input1",
        output="my model output1",
        started_at="2024-09-01T23:59:59",
        finished_at=datetime.now().astimezone(),
        annotation="Good",  # Either Good, Bad, Unknown, or None
        interaction_type="Generation",  # Optional. Defaults to the application's default type if not provided.
        session_id="session-1",  # Optional. Groups related interactions; auto-generated if not provided.
    )
    

• Session Support in Stream Upload

  • Added support for session_id in stream upload via the log_interaction method, facilitating real-time tracking of interactions within sessions.

    Python

    dc_client.log_interaction(
        app_name="DemoApp",
        version_name="v1",
        env_type=EnvType.EVAL,
        user_interaction_id="id-1",
        input="My Input",
        session_id="session-1",
        is_completed=False,
    )
    

• Session-Based Filtering in get_data

  • Enhanced the get_data method to include filtering by session_ids, providing greater flexibility in retrieving session-specific data.

    Python

    dc_client.get_data(
        app_name="MyAppName",
        version_name="MyVersionName",
        environment=EnvType.EVAL,
        session_ids=["session-1", "session-2"],
    )