This guide outlines how to integrate Deepchecks LLM Evaluation with your Anthropic models to monitor and analyze their performance.


Before you begin, ensure you have the following:

  • A Deepchecks LLM Evaluation account.
  • An Anthropic API key.
  • Python environment with the deepchecks-llm-client and anthropic packages installed (pip install deepchecks-llm-client anthropic).

Integration Steps

  1. Initialize Deepchecks Client:
from deepchecks_llm_client.client import dc_client
from deepchecks_llm_client.data_types import EnvType

    env_type=EnvType.EVAL,  # Change to EnvType.PROD for production monitoring
  1. Log Interactions with Anthropic Models:

Here's an example of how to log interactions with Anthropic's Claude model:

from deepchecks_llm_client.data_types import LogInteractionType, AnnotationType
from anthropic import Anthropic

# Configure Anthropic client
client = Anthropic(api_key="YOUR_ANTHROPIC_API_KEY")

def log_anthropic_interaction(user_input):
    # Make prediction using Anthropic model
    message = client.messages.create(
        messages=[{"role": "user", "content": user_input}],
    prediction = message.content

    # Log interaction to Deepchecks
        annotation=AnnotationType.UNKNOWN,  # Add annotation if available

# Example usage
user_input = "Write a short story about a robot who wants to become human."

This code snippet demonstrates how to:

  • Use the anthropic library to interact with Anthropic models.
  • Make predictions using the model.
  • Log the interaction data (input, output) to Deepchecks using the log_interaction method.
  1. View Insights in Deepchecks Dashboard:
    Once you've logged interactions, head over to the Deepchecks LLM Evaluation dashboard to analyze your model's performance. You can explore various insights, compare versions, and monitor production data.

Advanced Options

Deepchecks offers several advanced features for fine-grained control and analysis:

  • Updating Annotations and Custom Properties: You can update annotations and custom properties for logged interactions.
  • Logging Steps: For complex LLM pipelines, you can log individual steps with their inputs and outputs.
  • Additional Interaction Data: Log additional data like timestamps, user IDs, and custom properties for richer analysis.