Deepchecks in AWS SageMaker

AWS SageMaker Partner AI Apps

As a leading GenAI evaluation tool, Deepchecks was chosen as one of four partner applications for native integration with AWS SageMaker. Partner AI apps enables seamless operation entirely within AWS infrastructure, ensuring secure and efficient AI model evaluation. Customers benefit from combining AWS's scalable platform with Deepchecks' comprehensive evaluation tools, enhancing the reliability and precision of their AI applications across various use cases. This powerful collaboration supports businesses in achieving optimal performance and robustness in their AI deployments. Read more about Partner AI Apps in the AWS docs .

Running Deepchecks in AWS SageMaker

AWS Account Setup

Ensure you have an active AWS account (or watch this video for a guide on setting one up) and enter AWS SageMaker .

Deepchecks Setup in SageMaker

Follow the Partner AI Apps Set up Guide for setting up the relevant administrative and user permissions, and verify you have a Deepchecks subscription via SageMaker Partner AI Apps (free trials available).

AWS Account and SageMaker Setup:

Once your organization’s administrator sets up Deepchecks, users can access the platform through their AWS users. Team members can then access to the necessary tools via Amazon SageMaker AI and Amazon SageMaker Unified Studio. These integrated environments allow you to seamlessly leverage Partner AI Apps. To launch your SageMaker environment for GenAI, follow these steps:

• Access the Console: Log in to the AWS Management Console using your AWS credentials.
• Search for SageMaker: In the top search bar, type “SageMaker.” As you type, the console will display a filtered list of related services (e.g., “Amazon SageMaker” and “Amazon SageMaker Studio”).
• Select the Appropriate Service: Click on “Amazon SageMaker AI” or ”Amazon SageMaker” to navigate to its dedicated page.
• Provisioned Environment: Your organization’s administrator will have pre-provisioned the SageMaker Studio domain. This setup allows you and your team to immediately begin development with the Partner AI Apps deployed within the secure SageMaker ecosystem.
• Launch Studio Environment: Launch the Studio environment with the respective domain and user profile.

📘

Note

A "Deployed" status on Deepchecks for a Partner AI App indicates that the application has been successfully deployed in the SageMaker environment, is properly connected, and is ready to leverage its capabilities. Please note the Amazon Resource Name (ARN) and the Deepchecks Software Development Kit (SDK) URL, which are displayed when you open the Deepchecks Partner AI App.

• Get Deepchecks API: With the Deepchecks’ Partner AI Apps launch page open, click “Open Deepchecks LLM Evaluation”. This action launches a private instance of the Deepchecks LLM evaluation application within SageMaker. Next, generate and copy your Deepchecks API key alongside the ARN and SDK URL (AWS Partner App URL) required to integrate Deepchecks LLM Eval to log and analyze the performance of our RAG pipeline.

• Run JupyterLab: Now that your SageMaker environment is up and running, launch a JupyterLab workspace within SageMaker to create a notebook instance that can be shared with your team or used privately.

  • To get easily started with your first demo application, select one of the Example notebooks from the bottom section of the Deepchecks App page on SageMaker: