AWS Cloudwatch Integration

DeepChecks can be configured to send monitoring data and LLM evaluation metrics directly to your AWS CloudWatch account. This allows you to consolidate your model-monitoring data in one place, using your existing AWS dashboards and alarms.
This guide will walk you through creating the necessary IAM Role in your AWS account and providing its ARN (Amazon Resource Name) to DeepChecks.

Setup

The setup process involves creating an IAM role in your AWS account that DeepChecks can assume. This role will be granted a specific, limited policy that only allows it to write metric data to a designated CloudWatch namespace.

Part 1: Create the IAM Role

1. Sign in to your AWS account and navigate to the IAM service.

   

2. Go to Roles in the left-hand menu and click Create role.

   

3. For Trusted entity type, select Custom Trust Policy.

   

4. Paste the following trust policy:

   JSON

   {  
   	"Version": "2012-10-17",  
   	"Statement": \[  
   		{  
   			"Effect": "Allow",  
   			"Principal": {  
   				"AWS": "arn:aws:iam::TDB:root"  
   			},  
   			"Action": [  
   				"sts:AssumeRole",  
   				"sts:TagSession"  
   			],  
   			"Condition": {}  
   		}  
   	]  
   }

5. Click Next.

   

6. On the Add permissions page, click Next without adding any policies. We will add a specific inline policy in the next part to ensure the permissions are strictly limited.

7. On the Name, review, and create page:

   1. Role name: Enter a descriptive name, such as DeepchecksMetricsNotifierRole.

   2. Description (optional): Add a description, like "Allows DeepChecks to send metrics to CloudWatch."

      

   3. Review the details and click Create role.

      

Part 2: Add Permissions Policy

Now that the role is created, you need to attach the specific permissions policy that allows it to send metrics.

1. From your IAM Roles list, find and click on the role you just created (e.g., DeepchecksMetricsNotifierRole).

2. On the role's summary page, ensure you are on the Permissions tab.

3. Click the Add permissions button and Create inline policy.

   

4. This will open the policy editor. Select the JSON tab.

5. Delete any existing content in the editor and paste the following policy. This policy is strictly limited to the cloudwatch:PutMetricData action and only for the DeepChecksLLM namespace.

   JSON

   {  
       "Version": "2012-10-17",  
       "Statement": [  
           {  
               "Sid": "AllowSendCloudWatchMetrics",  
               "Effect": "Allow",  
               "Action": "cloudwatch:PutMetricData",  
               "Resource": "*",  
               "Condition": {  
                   "StringEquals": {  
                       "cloudwatch:namespace": "DeepChecksLLM"  
                   }  
               }  
           }  
       ]  
   }

6. Click Next.

   

7. Give the policy a Name, such as DeepChecksCloudWatchPolicy.

8. Click Create policy.

   

Part 3: Provide the Role ARN to DeepChecks

You have now successfully created a role and attached the necessary permissions. The final step is to provide the Role ARN to DeepChecks.

1. You should be back on the summary page for your DeepchecksMetricsNotifierRole.

2. At the top of the Summary section, find the ARN. It will look similar to arn:aws:iam::<Your-Account-ID>:role/DeepchecksMetricsNotifierRole.

3. Click the copy icon to copy the ARN to your clipboard.

   

4. Provide this Role ARN to your DeepChecks Solution Engineer.