AWS Cloudwatch Integration

Deepchecks can be configured to send monitoring data and LLM evaluation metrics directly to your AWS account CloudWatch. This allows you to consolidate your model-monitoring data in one place, using your existing AWS dashboards and alarms.
This guide will walk you through creating the necessary IAM permission.

Providing SageMakerAI permission to send your CloudWatch Metrics

In order to add the relevant permissions to the role used by AWS SageMaker to make the relevant API calls to CloudWatch, the following policy needs to be added to the execution role that you deployed Deepchecks with. (called PartnerAiAppExecutionRole in the AWS documentation).

{  
    "Version": "2012-10-17",  
    "Statement": [  
        {  
            "Sid": "AllowSendCloudWatchMetrics",  
            "Effect": "Allow",  
            "Action": "cloudwatch:PutMetricData",  
            "Resource": "*",  
            "Condition": {  
                "StringEquals": {  
                    "cloudwatch:namespace": "DeepChecksLLM"  
                }  
            }  
        }  
    ]  
}

Enabling the CloudWatch metrics feature

🔒
This is only accessible to users with the OWNER role

Navigate to Workspace Settings / Preference
Under Application Settings / Category select: CLOUDWATCH_SYSTEM_CONFIG

set enabled to true

📘
The role_arn and region_name options should be left as is.

📘
Due to a limitation of AWS SageMakerAI metrics can only be sent to the same region where the application is deployed.