Data Types

Type definitions used throughout the Deepchecks LLM Python SDK. Enums describe categorical fields accepted by the client; dataclasses describe structured payloads sent to or returned from the API.

Enums

AnnotationType

enum AnnotationType(value)

Member Type: str

Human annotation label for an interaction (good, bad, or unknown).

Valid values are as follows:

• GOOD = <AnnotationType.GOOD: 'good'>
• BAD = <AnnotationType.BAD: 'bad'>
• UNKNOWN = <AnnotationType.UNKNOWN: 'unknown'>

The Enum and its members also have the following methods:

__new__(value)

ApplicationType

enum ApplicationType(value)

Member Type: str

Category of an LLM application (e.g. Q&A, Chat, Summarization, Agent).

Valid values are as follows:

• QA = <ApplicationType.QA: 'Q&A'>
• OTHER = <ApplicationType.OTHER: 'Other'>
• SUMMARIZATION = <ApplicationType.SUMMARIZATION: 'Summarization'>
• CLASSIFICATION = <ApplicationType.CLASSIFICATION: 'Classification'>
• GENERATION = <ApplicationType.GENERATION: 'Generation'>
• FEATURE_EXTRACTION = <ApplicationType.FEATURE_EXTRACTION: 'Feature Extraction'>
• RETRIEVAL = <ApplicationType.RETRIEVAL: 'Retrieval'>
• CHAT = <ApplicationType.CHAT: 'Chat'>
• CHAIN = <ApplicationType.CHAIN: 'Chain'>
• ROOT = <ApplicationType.ROOT: 'Root'>
• LLM = <ApplicationType.LLM: 'LLM'>
• AGENT = <ApplicationType.AGENT: 'Agent'>
• TOOL = <ApplicationType.TOOL: 'Tool'>

The Enum and its members also have the following methods:

__new__(value)

BuiltInInteractionType

enum BuiltInInteractionType(value)

Member Type: str

Predefined interaction type categories available for classifying application interactions.

Valid values are as follows:

• QA = <BuiltInInteractionType.QA: 'Q&A'>
• OTHER = <BuiltInInteractionType.OTHER: 'Other'>
• SUMMARIZATION = <BuiltInInteractionType.SUMMARIZATION: 'Summarization'>
• CLASSIFICATION = <BuiltInInteractionType.CLASSIFICATION: 'Classification'>
• GENERATION = <BuiltInInteractionType.GENERATION: 'Generation'>
• FEATURE_EXTRACTION = <BuiltInInteractionType.FEATURE_EXTRACTION: 'Feature Extraction'>
• RETRIEVAL = <BuiltInInteractionType.RETRIEVAL: 'Retrieval'>
• CHAT = <BuiltInInteractionType.CHAT: 'Chat'>
• CHAIN = <BuiltInInteractionType.CHAIN: 'Chain'>
• ROOT = <BuiltInInteractionType.ROOT: 'Root'>
• LLM = <BuiltInInteractionType.LLM: 'LLM'>
• AGENT = <BuiltInInteractionType.AGENT: 'Agent'>
• TOOL = <BuiltInInteractionType.TOOL: 'Tool'>

The Enum and its members also have the following methods:

__new__(value)

CategoryName

enum CategoryName(value)

Member Type: str

Configuration categories supported by the unified /configs endpoint.

Mirrors deepchecks_llm_eval.utils.settings_store.categories.CategoryName
on the backend; only the customer-facing subset is exposed here.

Valid values are as follows:

• BATCHER = <CategoryName.BATCHER: 'BATCHER'>
• SAMPLING = <CategoryName.SAMPLING: 'SAMPLING'>
• TOPICS = <CategoryName.TOPICS: 'TOPICS'>
• LLM_TRANSLATION = <CategoryName.LLM_TRANSLATION: 'LLM_TRANSLATION'>
• LLM_TRANSLATION_DETECTION = <CategoryName.LLM_TRANSLATION_DETECTION: 'LLM_TRANSLATION_DETECTION'>
• LLM_CONTEXT_CLASSIFICATION_MODEL = <CategoryName.LLM_CONTEXT_CLASSIFICATION_MODEL: 'LLM_CONTEXT_CLASSIFICATION_MODEL'>
• LLM_PROPERTIES_MODEL = <CategoryName.LLM_PROPERTIES_MODEL: 'LLM_PROPERTIES_MODEL'>
• LLM_WEAK_MODEL = <CategoryName.LLM_WEAK_MODEL: 'LLM_WEAK_MODEL'>
• LLM_STRONG_MODEL = <CategoryName.LLM_STRONG_MODEL: 'LLM_STRONG_MODEL'>

The Enum and its members also have the following methods:

__new__(value)

DatasetType

enum DatasetType(value)

Member Type: str

Enum for dataset types.

Valid values are as follows:

• SINGLE_TURN = <DatasetType.SINGLE_TURN: 'SINGLE_TURN'>
• MULTI_TURN = <DatasetType.MULTI_TURN: 'MULTI_TURN'>

The Enum and its members also have the following methods:

__new__(value)

EnvType

enum EnvType(value)

Member Type: str

Environment type for logging interactions (production, evaluation, or penetration testing).

Valid values are as follows:

• PROD = <EnvType.PROD: 'PROD'>
• EVAL = <EnvType.EVAL: 'EVAL'>
• PENTEST = <EnvType.PENTEST: 'PENTEST'>

The Enum and its members also have the following methods:

__new__(value)

InteractionCompleteEvents

enum InteractionCompleteEvents(value)

Member Type: str

Server-side processing events that fire when an interaction's evaluation pipeline completes a stage.

Valid values are as follows:

• TOPICS_COMPLETED = <InteractionCompleteEvents.TOPICS_COMPLETED: 'topics_completed'>
• PROPERTIES_COMPLETED = <InteractionCompleteEvents.PROPERTIES_COMPLETED: 'properties_completed'>
• SIMILARITY_COMPLETED = <InteractionCompleteEvents.SIMILARITY_COMPLETED: 'similarity_completed'>
• LLM_PROPERTIES_COMPLETED = <InteractionCompleteEvents.LLM_PROPERTIES_COMPLETED: 'llm_properties_completed'>
• ANNOTATION_COMPLETED = <InteractionCompleteEvents.ANNOTATION_COMPLETED: 'annotation_completed'>
• DC_EVALUATION_COMPLETED = <InteractionCompleteEvents.DC_EVALUATION_COMPLETED: 'dc_evaluation_completed'>
• BUILTIN_LLM_PROPERTIES_COMPLETED = <InteractionCompleteEvents.BUILTIN_LLM_PROPERTIES_COMPLETED: 'builtin_llm_properties_completed'>

The Enum and its members also have the following methods:

__new__(value)

LLMModelCategory

enum LLMModelCategory(value)

Member Type: str

LLM model category, kept as a thin alias over :class:CategoryName.

Valid values are as follows:

• BASIC = <LLMModelCategory.BASIC: 'CategoryName.LLM_WEAK_MODEL.value'>
• ADVANCED = <LLMModelCategory.ADVANCED: 'CategoryName.LLM_STRONG_MODEL.value'>
• LLM_PROPERTIES = <LLMModelCategory.LLM_PROPERTIES: 'CategoryName.LLM_PROPERTIES_MODEL.value'>

The Enum and its members also have the following methods:

__new__(value)

PropertyColumnType

enum PropertyColumnType(value)

Member Type: str

Column type for user-defined properties (categorical or numeric).

Valid values are as follows:

• CATEGORICAL = <PropertyColumnType.CATEGORICAL: 'categorical'>
• NUMERIC = <PropertyColumnType.NUMERIC: 'numeric'>

The Enum and its members also have the following methods:

__new__(value)

SpanKind

enum SpanKind(value)

Member Type: str

Classification of a span's role within a trace (LLM call, tool use, chain, agent, or retrieval).

Valid values are as follows:

• LLM = <SpanKind.LLM: 'LLM'>
• TOOL = <SpanKind.TOOL: 'TOOL'>
• CHAIN = <SpanKind.CHAIN: 'CHAIN'>
• AGENT = <SpanKind.AGENT: 'AGENT'>
• RETRIEVAL = <SpanKind.RETRIEVAL: 'RETRIEVER'>

The Enum and its members also have the following methods:

__new__(value)

Dataclasses

AppSamplingConfig

class AppSamplingConfig

Attributes: prod_sampling_ratio

__init__(prod_sampling_ratio: float | None = None) → None

Application

class Application

Attributes: description, log_latest_insert_time_epoch, n_of_llm_properties, n_of_interactions, notifications_enabled

__init__(id: int, name: str, kind: ApplicationType, created_at: datetime, updated_at: datetime, versions: List[ApplicationVersion], interaction_types: List[str], description: str | None = None, log_latest_insert_time_epoch: int | None = None, n_of_llm_properties: int | None = None, n_of_interactions: int | None = None, notifications_enabled: bool | None = None) → None

A registered LLM application with its versions, interaction types, and configuration.

ApplicationVersion

class ApplicationVersion

Attributes: description, additional_fields

__init__(id: int, name: str, created_at: datetime, updated_at: datetime, description: str | None = None, additional_fields: Dict[str, Any] | None = None) → None

A dataclass representing an Application Version.

ApplicationVersionSchema

class ApplicationVersionSchema

Attributes: description, additional_fields

__init__(name: str, description: str | None = None, additional_fields: Dict[str, Any] | None = None) → None

Schema for creating a new application version with a name and optional metadata.

BatcherConfig

class BatcherConfig

Attributes: batch_size, interval_seconds, should_wait_between_batches

__init__(batch_size: int | None = None, interval_seconds: int | None = None, should_wait_between_batches: bool | None = None) → None

Workspace-level batcher configuration.

Mirrors the backend BatcherConfigData settings model. Callers may set
any combination of the three fields when updating; omitted values keep
their current setting.

CreateInteractionTypeVersionData

class CreateInteractionTypeVersionData

Attributes: model, prompt, metadata_params

__init__(interaction_type_id: int, application_version_id: int, model: str | None = None, prompt: str | None = None, metadata_params: Dict[str, Any] = None) → None

A dataclass for creating interaction type version data.

Dataset

class Dataset

Attributes: dataset_type

__init__(id: int, application_id: int, dataset_name: str, samples_count: int, created_at: datetime, updated_at: datetime, dataset_type: DatasetType = DatasetType.SINGLE_TURN) → None

Data class representing a dataset.

DatasetRunResult

class DatasetRunResult

Attributes: app_name, version_name, env_type, additional_headers, verify_ssl, iteration

__init__(dataset_name: str, deployment_name: str, total_samples: int, successful_samples: int, failed_samples: int, duration_seconds: float, results: List[DatasetRunSampleResult], app_name: str | None = None, version_name: str | None = None, env_type: str | None = None, additional_headers: Dict[str, str] | None = None, verify_ssl: bool = True, iteration: int = 1) → None

Result of running a dataset on a deployment.

DatasetRunSampleResult

class DatasetRunSampleResult

Attributes: deployment_response, error_message, error_type, duration_seconds, retries, status_code

__init__(sample_id: int, sample_input: Any, sample_output: Any | None, sample_metadata: Dict[str, Any] | None, success: bool, deployment_response: Dict[str, Any] | None = None, error_message: str | None = None, error_type: str | None = None, duration_seconds: float | None = None, retries: int = 0, status_code: int | None = None) → None

Result of running a single dataset sample.

DatasetSample

class DatasetSample

Attributes: created_at, updated_at

__init__(id: int, dataset_id: int, input: Any, output: Any | None, sample_metadata: Dict[str, Any] | None, created_at: datetime | None = None, updated_at: datetime | None = None) → None

Data class representing a dataset sample.

Deployment

class Deployment

__init__(id: int, application_id: int, deployment_name: str, deployment_url: str, timeout: int, max_concurrent: int, max_retries: int, created_at: datetime, updated_at: datetime, headers: List[DeploymentHeader]) → None

Data class representing a deployment configuration.

DeploymentHeader

class DeploymentHeader

__init__(id: int, deployment_id: int, name: str, value: str) → None

Data class representing a deployment header.

DocClassificationConfig

class DocClassificationConfig

__init__(enabled: bool) → None

Interaction

class Interaction

__init__(user_interaction_id: str | int, input: str, output: str, information_retrieval: str | List[str], history: str | List[str], full_prompt: str, expected_output: str, is_completed: bool, metadata: Dict[str, str], tokens: int, input_tokens: int | None, output_tokens: int | None, model: str | None, model_provider: str | None, input_cost: float | None, output_cost: float | None, cost: float | None, properties: Dict[str, Any], properties_reasons: Dict[str, Any], created_at: datetime, interaction_datetime: datetime, interaction_type: str, topic: str, session_id: str | int, annotation: AnnotationType | None, annotation_reason: str | None) → None

A read-only representation of a logged interaction returned by the API.

InteractionType

class InteractionType

__init__(id: int, name: str) → None

An interaction type definition consisting of an ID and name.

InteractionTypeVersionData

class InteractionTypeVersionData

Attributes: model, prompt, metadata_params, created_at, updated_at

__init__(id: int, interaction_type_id: int, application_version_id: int, model: str | None = None, prompt: str | None = None, metadata_params: Dict[str, Any] = None, created_at: datetime = None, updated_at: datetime = None) → None

A dataclass representing interaction type version data.

InteractionUpdate

class InteractionUpdate

Attributes: input, output, information_retrieval, history, full_prompt, expected_output, is_completed, metadata, tokens, input_tokens, output_tokens, model, model_provider, annotation, annotation_reason, steps, user_value_properties, started_at, finished_at

__init__(input: str | None = None, output: str | None = None, information_retrieval: str | List[str] | None = None, history: str | List[str] | None = None, full_prompt: str | None = None, expected_output: str | None = None, is_completed: bool = True, metadata: Dict[str, str] | None = None, tokens: int | None = None, input_tokens: int | None = None, output_tokens: int | None = None, model: str | None = None, model_provider: str | None = None, annotation: AnnotationType | str | None = None, annotation_reason: str | None = None, steps: List[Step] | None = None, user_value_properties: List[UserValueProperty] | None = None, started_at: datetime | float | None = None, finished_at: datetime | float | None = None) → None

A dataclass representing an update interaction object.

LLMModelSettings

class LLMModelSettings

__init__(name: str, category: LLMModelCategory) → None

A configured default LLM model.

LogInteraction

class LogInteraction

Attributes: user_interaction_id, vuln_type, vuln_trigger_str, topic, interaction_type, interaction_type_id, session_id

__init__(user_interaction_id: str | int | None = None, vuln_type: str | None = None, vuln_trigger_str: str | None = None, topic: str | None = None, interaction_type: str | None = None, interaction_type_id: int | None = None, session_id: str | int | None = None) → None

A dataclass representing a new interaction object.

Span

class Span

Attributes: status_code, status_description, input, output, full_prompt, expected_output, information_retrieval, tokens, input_tokens, output_tokens, model, model_provider, graph_parent_name, session_id, metadata, events, user_value_properties, steps

__init__(span_id: str, span_name: str, trace_id: str, span_kind: SpanKind, parent_id: str | None, started_at: float, finished_at: float, status_code: Literal['OK', 'ERROR'] = 'OK', status_description: str | None = None, input: str | None = None, output: str | None = None, full_prompt: str | None = None, expected_output: str | None = None, information_retrieval: List[str] | None = None, tokens: int | None = None, input_tokens: int | None = None, output_tokens: int | None = None, model: str | None = None, model_provider: str | None = None, graph_parent_name: str | None = None, session_id: str | None = None, metadata: Dict[str, str] | None = None, events: List[SpanEvent] | None = None, user_value_properties: List[UserValueProperty] | None = None, steps: List[Step] | None = None) → None

A dataclass representing a span within a trace for tracking nested operations.

A Span represents a unit of work within a distributed trace, allowing you to track
hierarchical relationships between operations. This is designed to work with the
OtelParser system for converting spans into interactions.

SpanEvent

class SpanEvent

Attributes: attributes

__init__(name: str, timestamp: float, attributes: Dict[str, Any] | None = None) → None

A dataclass representing an event that occurred during a span's execution.

Step

class Step

__init__(name: str, value: str) → None

A named key-value pair representing an intermediate step within an interaction or span.

TopicsConfig

class TopicsConfig

__init__(enabled: bool) → None

TranslationConfig

class TranslationConfig

Attributes: enabled, model_name, with_text_splitting

__init__(enabled: bool | None = None, model_name: str | None = None, with_text_splitting: bool | None = None) → None

Translation feature configuration for an application.

TranslationDetectionConfig

class TranslationDetectionConfig

Attributes: model_name

__init__(model_name: str | None = None) → None

Translation language-detection model configuration for an application.

UpdateInteractionTypeVersionData

class UpdateInteractionTypeVersionData

Attributes: model, prompt, metadata_params

__init__(model: str | None = None, prompt: str | None = None, metadata_params: Dict[str, Any] | None = None) → None

A dataclass for updating interaction type version data.

UserValueProperty

class UserValueProperty

Attributes: reason

__init__(name: str, value: Any, reason: str | None = None) → None

Data class representing user provided property

UserValuePropertyType

class UserValuePropertyType

Attributes: description

__init__(display_name: str, type: PropertyColumnType | str, description: str | None = None) → None

Schema definition for a user-provided property, specifying its display name and column type.