Production Monitoring
After customizing Deepchecks for a given LLM-based app, utilize the artifacts created to continuously monitor both quality and risk metrics in production
Production monitoring is an important capability for any software application. In the context of an LLM app, it’s both more important and more challenging than the “average” software application: The solution’s quality has many “fluctuations”, there are use cases that aren’t handled well that can become more common, and there is the possibility of changes related either to LLM APIs or the internal accompanying code and guardrails. And clearly, monitoring raw text is a challenging task.
Updated about 1 month ago
What’s Next