Documentation
DocumentationAPI ReferenceRelease Notes
DocumentationLog In
API Reference

DeepchecksLLMClient

Module containing all the client functionality for the Deepchecks LLM Client.

DeepchecksLLMClient

class DeepchecksLLMClient

DeepchecksLLMClient is the class through which you can interact with the Deepchecks LLM API.

Constructor

init

def __init__(host: str = None,
             api_token: str = None,
             log_level: int = logging.WARNING,
             silent_mode: bool = False) -> None

Init Deepchecks LLM Client. Connect to the server and perform a simple handshake with the API.

Parameters
  • host (str, default=None) — The URL of the Deepchecks server to communicate with. By default, it's set to the Deepchecks LLM host (https://app.llm.deepchecks.com), but you can override it by providing a custom URL. If not provided, the value is read from the environment variable DEEPCHECKS_LLM_HOST.
  • api_token (str, default=None) — Deepchecks API Token (can be generated from the UI). If not provided, the value is read from the environment variable DEEPCHECKS_LLM_API_TOKEN.
  • log_level (int, default=logging.WARNING) — Set SDK loggers logging level
  • silent_mode (bool, default=False) — If True, the SDK will print logs upon encountering errors rather than raising them
Returns

Attributes

AttributeDescription
apiGet the API object for the client.

Methods

MethodDescription
add_dataset_samplesAdd samples to a dataset.
annotateAnnotate a specific interaction by its user_interaction_id.
create_app_versionCreate a new application version.
create_applicationCreate a new application.
create_datasetCreate a new dataset.
create_deploymentCreate a new deployment.
create_interaction_typeActivating Builtin type or Creating a new custom interaction type for the current application.
create_interaction_type_version_dataCreate interaction type version data.
create_user_value_propertyDefine a user value property for the current application.
data_iteratorFetch all interactions from the specified environment type (PROD/EVAL) as an iterable.
delete_interaction_typeDelete interaction type.
delete_interactionsDelete specific interactions by their user_interaction_ids.
execute_appRun a dataset on a deployment with parallel execution.
get_all_dataset_samplesGet all samples from a dataset (handles pagination automatically).
get_app_basic_advanced_llm_modelsGet the per-app Basic and Advanced LLM models.
get_app_default_llm_modelGet the per-app default LLM model used for property evaluation.
get_app_document_classification_configGet the per-app document classification configuration.
get_app_sampling_configGet the per-app production sampling configuration.
get_app_topics_configGet the per-app topics configuration.
get_app_translation_configGet the per-app translation configuration.
get_app_translation_detection_configGet the per-app translation language-detection model configuration.
get_applicationsGet all applications as Application objects.
get_batcher_configGet the workspace batcher configuration.
get_dataFetch all the interactions from the specified environment (PROD/EVAL) as a pandas DataFrame.
get_data_if_calculations_completedCheck if calculations for specified events are completed by calling get_interactions_complete_status.
get_datasetGet a specific dataset by name.
get_dataset_samplesGet samples from a dataset with pagination.
get_datasetsGet all datasets for an application.
get_deploymentGet a specific deployment by name.
get_deploymentsGet all deployments for an application.
get_interaction_by_user_interaction_idGet a specific interaction by its user_interaction_id.
get_interaction_type_configWrite the auto-annotation YAML configuration file to the specified path.
get_interaction_type_version_dataGet interaction type version data.
get_interaction_typesGet the custom interaction types defined for the current application.
get_interactions_complete_statusGet the completion status of interactions for specified events and user interaction IDs.
get_pentest_promptsGet pentest prompts for the application.
get_raw_spansDownload raw OTEL spans matching the filter.
get_user_value_propertiesGet the user value properties defined for the current application.
get_versionsGet all versions of an application as Version objects.
get_workspace_llm_default_modelGet the workspace-level default LLM model used for property evaluation.
log_batch_interactionsLog multiple interactions at once.
log_interactionLog a single interaction.
log_spansSend spans to the Deepchecks LLM backend.
log_spans_fileSend spans to the Deepchecks LLM backend.
set_session_property_valuesSet user value properties on a session.
update_app_basic_advanced_llm_modelsSet the per-app Basic and/or Advanced LLM models.
update_app_default_llm_modelSet the per-app default LLM model used for property evaluation.
update_app_document_classification_configEnable or disable per-app document classification.
update_app_sampling_configUpdate the per-app production sampling ratio.
update_app_topics_configEnable or disable per-app topics generation.
update_app_translation_configUpdate the per-app translation configuration.
update_app_translation_detection_configSet the per-app translation language-detection model.
update_application_versionCreate a new application.
update_batcher_configUpdate the workspace batcher configuration.
update_datasetUpdate a dataset's name.
update_deploymentUpdate a deployment.
update_interactionUpdate a specific interaction by its user_interaction_id.
update_interaction_typeUpdate interaction type for the current application.
update_interaction_type_configUpdate the auto-annotation YAML configuration file for the interaction type within the application.
update_workspace_llm_default_modelSet the workspace-level default LLM model used for property evaluation.

api

@property
def api() -> API

Get the API object for the client.

Returns
  • API

add_dataset_samples

def add_dataset_samples(app_name: str,
                        dataset_name: str,
                        samples: List[Dict[str, Any]]) -> None

Add samples to a dataset.

Parameters
  • app_name (str) — Application name
  • dataset_name (str) — Dataset name
  • samples (List[Dict[str, Any]]) — List of samples to add. Each sample should have: - input: dict (required) - output: dict (optional) - sample_metadata: dict (optional)
Returns

annotate

def annotate(app_name: str,
             version_name: str,
             user_interaction_id: str,
             annotation: Union[AnnotationType, str] = None,
             reason: Optional[str] = None) ->     None

Annotate a specific interaction by its user_interaction_id.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of the version to which this interaction belongs
  • user_interaction_id (str) — The user_interaction_id of the interaction to annotate
  • annotation (AnnotationType | str, default=None) — Could be one of AnnotationType.GOOD, AnnotationType.BAD, AnnotationType.UNKNOWN or None to remove annotation
  • reason (str | None, default=None) — String that explains the reason for the annotation
Returns
    • None*

create_app_version

def create_app_version(app_name: str,
                       version_name: str,
                       description: Optional[str] = None,
                       additional_fields: Optional[Dict[str, Any]] = None) ->     The response from the API

Create a new application version.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Version name
  • description (str | None, default=None) — Description of the version
  • additional_fields (Dict[str, Any] | None, default=None) — Additional fields to add to the version
Returns
    • The response from the API*

create_application

def create_application(app_name: str,
                       app_type: ApplicationType,
                       versions: Optional[List[ApplicationVersionSchema]] = None,
                       description: Optional[str] = None) ->     The response from the API

Create a new application.

Parameters
  • app_name (str) — Application name
  • app_type (ApplicationType) — The Default Interaction Type for interactions of this application
  • versions (List[ApplicationVersionSchema] | None, default=None) — List of versions to create for the application
  • description (str | None, default=None) — Description of the application
Returns
    • The response from the API*

create_dataset

def create_dataset(app_name: str,
                   dataset_name: str,
                   dataset_type: DatasetType = DatasetType.SINGLE_TURN) -> Dataset

Create a new dataset.

Parameters
  • app_name (str) — Application name
  • dataset_name (str) — Name for the new dataset
  • dataset_type (DatasetType, default=DatasetType.SINGLE_TURN) — Type of dataset, defaults to DatasetType.SINGLE_TURN
Returns

create_deployment

def create_deployment(app_name: str,
                      deployment_name: str,
                      deployment_url: str,
                      timeout: int = 30,
                      max_concurrent: int = 5,
                      max_retries: int = 2,
                      headers: Optional[List[Dict[str, Any]]] = None) -> Deployment

Create a new deployment.

Parameters
  • app_name (str) — Application name
  • deployment_name (str) — Name for the deployment
  • deployment_url (str) — URL of the deployment endpoint
  • timeout (int, default=30) — Request timeout in seconds (5-300)
  • max_concurrent (int, default=5) — Maximum concurrent requests (1-20)
  • max_retries (int, default=2) — Maximum number of retries (0-5)
  • headers (List[Dict[str, Any]] | None, default=None) — List of headers to include. Each header should have: - name: str (required) - value: str (required)
Returns

create_interaction_type

def create_interaction_type(app_name: str,
                            name: str,
                            template_name: Optional[BuiltInInteractionType] = None) -> int

Activating Builtin type or Creating a new custom interaction type for the current application.

Parameters
  • app_name (str) — Application name
  • name (str) — Name of the interaction type
  • template_name (BuiltInInteractionType | None, default=None) — Template of the interaction type
Returns
  • int — Interaction type id

create_interaction_type_version_data

def create_interaction_type_version_data(app_name: str,
                                         version_name: str,
                                         interaction_type: str,
                                         model: Optional[str] = None,
                                         prompt: Optional[str] = None,
                                         metadata_params: Optional[Dict[str, Any]] = None) -> InteractionTypeVersionData

Create interaction type version data.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Version name
  • interaction_type (str) — Interaction type name
  • model (str | None, default=None) — Model name
  • prompt (str | None, default=None) — Prompt template
  • metadata_params (Dict[str, Any] | None, default=None) — Additional metadata parameters
Returns

create_user_value_property

def create_user_value_property(app_name: str,
                               name: str,
                               prop_type: Union[PropertyColumnType, str],
                               description: Optional[str] = None,
                               interaction_type: Optional[str] = None) -> None

Define a user value property for the current application.

Parameters
  • app_name (str) — Application name
  • name (str) — Name of the user value property
  • prop_type (PropertyColumnType | str) — Type of the user value property (categorical, numeric)
  • description (str | None, default=None) — Description of the user value property
  • interaction_type (str | None, default=None) — Interaction type name
Returns

data_iterator

def data_iterator(app_name: str,
                  version_name: str,
                  env_type: Union[EnvType, str],
                  start_time: Union[datetime, int, None] = None,
                  end_time: Union[datetime, int, None] = None,
                  user_interaction_ids: Union[List[Union[str, int]], None] = None,
                  include_incomplete: bool = False,
                  session_ids: Union[List[Union[str, int]], None] = None,
                  interaction_type: Union[str, None] = None) -> Iterable[Interaction]

Fetch all interactions from the specified environment type (PROD/EVAL) as an iterable.

Supports pagination, so this API is suitable for iterating over large amounts of data.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of application version
  • env_type (EnvType | str) — The environment type from which to fetch interactions. This can be either "PROD" or "EVAL".
  • start_time (datetime | int | None, default=None) — The start time from which to fetch interactions. This can be a datetime object or an integer. If not provided, interactions will be fetched from the beginning.
  • end_time (datetime | int | None, default=None) — The end time until which to fetch interactions. This can be a datetime object or an integer. If not provided, interactions will be fetched up to the most recent.
  • user_interaction_ids (List[str | int] | None, default=None) — user interactions ids to include into the list of interactions.annotation_reason
  • include_incomplete (bool, default=False) — Return only interactions with completed status or not. Only completed are returned by default.
  • session_ids (List[str | int] | None, default=None) — List of session IDs to filter by. Maximum 1000 items allowed.
  • interaction_type (str | None, default=None) — Interaction type name to filter by.
Returns

delete_interaction_type

def delete_interaction_type(app_name: str, interaction_type_id: int) -> None

Delete interaction type.

Parameters
  • app_name (str) — Application name
  • interaction_type_id (int) — Interaction type id
Returns

delete_interactions

def delete_interactions(app_name: str,
                        version_name: str,
                        user_interaction_ids: List[str]) ->     None

Delete specific interactions by their user_interaction_ids.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of application version
  • user_interaction_ids (List[str]) — List of interaction user ids to delete
Returns
    • None*

execute_app

def execute_app(app_name: str,
                dataset_name: str,
                deployment_name: str,
                version_name: Optional[str] = None,
                env_type: Optional[str] = None,
                additional_headers: Optional[Dict[str, str]] = None,
                show_progress: bool = True,
                verify_ssl: bool = True,
                progress_callback: Optional[Callable[[int, int], None]] = None,
                sample_callback: Optional[Callable] = None) -> DatasetRunResult

Run a dataset on a deployment with parallel execution.

This method fetches the dataset samples and deployment configuration,
then runs all samples against the deployment endpoint using the
configured parallelism and retry settings.

Note: This is an async method and must be called with await:

>>> result = await client.execute_app("my-app", "test-dataset", "prod-deployment")
Parameters
  • app_name (str) — Application name
  • dataset_name (str) — Dataset name to run
  • deployment_name (str) — Deployment name to run against
  • version_name (str | None, default=None) — Version name to include in dc_fields
  • env_type (str | None, default=None) — Environment type to include in dc_fields
  • additional_headers (Dict[str, str] | None, default=None) — Additional headers to include in requests (beyond deployment headers)
  • show_progress (bool, default=True) — Whether to show a live progress widget (if in notebook with ipywidgets installed)
  • verify_ssl (bool, default=True) — Whether to verify SSL certificates
  • progress_callback (Callable[[int, int], None] | None, default=None) — Callback function that receives (completed_count, total_count) after each sample completes
  • sample_callback (Callable | None, default=None) — Callback function that receives (sample_id, status, keyword args) for each sample status update. Keyword args may include: input, output, error, duration, retries
Returns
  • DatasetRunResult — Results of running the dataset, including success/failure counts, individual sample results, and timing information.
Examples
>>> result = await client.execute_app("my-app", "test-dataset", "prod-deployment")
>>> print(f"Success rate: {result.success_rate:.1f}%")
>>> print(f"Failed samples: {result.failed_samples}")
>>> # Re-run only failed samples
>>> if result.failed_samples > 0:
...     retry_result = await result.rerun_failed()
...     print(f"Retry success rate: {retry_result.success_rate:.1f}%")

get_all_dataset_samples

def get_all_dataset_samples(app_name: str, dataset_name: str) -> List[DatasetSample]

Get all samples from a dataset (handles pagination automatically).

Parameters
  • app_name (str) — Application name
  • dataset_name (str) — Dataset name
Returns

get_app_basic_advanced_llm_models

def get_app_basic_advanced_llm_models(app_name: str) -> list of LLMModelSettings

Get the per-app Basic and Advanced LLM models.

Parameters
  • app_name (str) — Application name.
Returns
  • list of LLMModelSettings — Two-element list with the Basic entry first and the Advanced entry second.

get_app_default_llm_model

def get_app_default_llm_model(app_name: str) -> LLMModelSettings

Get the per-app default LLM model used for property evaluation.

Parameters
  • app_name (str) — Application name.
Returns
  • LLMModelSettings — The application-level LLM_PROPERTIES model. Inherits from the workspace default if no app-specific override is set.

get_app_document_classification_config

def get_app_document_classification_config(app_name: str) -> DocClassificationConfig

Get the per-app document classification configuration.

Parameters
  • app_name (str) — Application name.
Returns

get_app_sampling_config

def get_app_sampling_config(app_name: str) -> AppSamplingConfig

Get the per-app production sampling configuration.

Parameters
  • app_name (str) — Application name.
Returns
  • AppSamplingConfig — The current production sampling ratio (0..1) for this application.

get_app_topics_config

def get_app_topics_config(app_name: str) -> TopicsConfig

Get the per-app topics configuration.

Parameters
  • app_name (str) — Application name.
Returns
  • TopicsConfig — enabled: whether topics generation is enabled for this app.

get_app_translation_config

def get_app_translation_config(app_name: str) -> TranslationConfig

Get the per-app translation configuration.

Parameters
  • app_name (str) — Application name.
Returns
  • TranslationConfig — enabled, model_name, and with_text_splitting flags currently set for this application.

get_app_translation_detection_config

def get_app_translation_detection_config(app_name: str) -> TranslationDetectionConfig

Get the per-app translation language-detection model configuration.

Parameters
  • app_name (str) — Application name.
Returns

get_applications

def get_applications() -> List[Application]

Get all applications as Application objects.

Note that "custom" attribute in ApplicationVersion is deprecated. Please use additional_fields instead.

Returns

get_batcher_config

def get_batcher_config() -> BatcherConfig

Get the workspace batcher configuration.

Returns
  • BatcherConfig — The current batcher settings (batch size, interval, and whether the batcher waits between batches).

get_data

def get_data(app_name: str,
             version_name: str,
             env_type: Union[EnvType, str],
             return_topics: bool = True,
             return_annotation_data: bool = True,
             return_builtin_props: bool = True,
             return_user_value_properties: bool = True,
             return_custom_prompt: bool = True,
             start_time: Union[datetime, int, None] = None,
             end_time: Union[datetime, int, None] = None,
             return_similarities: bool = False,
             user_interaction_ids: Union[List[str], None] = None,
             include_incomplete: bool = False,
             return_steps: bool = True,
             session_ids: Union[List[str], None] = None,
             interaction_type: Union[str, None] = None,
             return_session_related: bool = False,
             return_only_input_output_texts: bool = False) -> DataFrame | None

Fetch all the interactions from the specified environment (PROD/EVAL) as a pandas DataFrame.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of application version
  • env_type (EnvType | str) — The environment type from which to fetch interactions. This can be either "PROD" or "EVAL".
  • return_topics (bool, default=True) — Whether to include the topic in the data.
  • return_annotation_data (bool, default=True) — Whether to include annotation info in the data.
  • return_builtin_props (bool, default=True) — Whether to include built-in properties in the data.
  • return_user_value_properties (bool, default=True) — Whether to include user value properties in the data.
  • return_custom_prompt (bool, default=True) — Whether to include LLM properties in the data.
  • start_time (datetime | int | None, default=None) — The start time from which to fetch interactions. This can be a datetime object or an integer. If not provided, interactions will be fetched from the beginning.
  • end_time (datetime | int | None, default=None) — The end time until which to fetch interactions. This can be a datetime object or an integer. If not provided, interactions will be fetched up to the most recent.
  • return_similarities (bool, default=False) — Whether to include similarities in the data.
  • user_interaction_ids (List[str] | None, default=None) — user interactions ids to include in the data.
  • include_incomplete (bool, default=False) — Return only interactions with completed status or not. Only completed are returned by default.
  • return_steps (bool, default=True) — Whether to include steps in the data.
  • session_ids (List[str] | None, default=None) — List of session IDs to filter by. Maximum 1000 items allowed.
  • interaction_type (str | None, default=None) — interaction type to filter by.
  • return_session_related (bool, default=False) — Whether to include session related data in the data.
  • return_only_input_output_texts (bool, default=False) — If true the only text fields that would be included are input and output.
Returns
  • DataFrame | None — A pandas DataFrame containing the interactions, or None in case of a problem retrieving the data.

get_data_if_calculations_completed

def get_data_if_calculations_completed(app_name: str,
                                       version_name: str,
                                       env_type: EnvType,
                                       user_interaction_ids: List[str],
                                       events_to_check: Optional[List[InteractionCompleteEvents]] = None,
                                       max_retries: int = 60,
                                       retry_interval_in_seconds: int = 3,
                                       return_topics: bool = True,
                                       return_annotation_data: bool = True,
                                       return_builtin_props: bool = True,
                                       return_user_value_properties: bool = True,
                                       return_custom_prompt: bool = True,
                                       return_similarities: bool = False) -> Optional[DataFrame]

Check if calculations for specified events are completed by calling get_interactions_complete_status.

If all user_interaction_ids completed then we will fetch the data for those ids,
if not - will sleep and retry again. Consider increasing max_retries or retry_interval_in_seconds if running
the method returns None.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of application version
  • env_type (EnvType) — Environment type
  • user_interaction_ids (List[str]) — A list of user interaction IDs to retrieve data for.
  • events_to_check (List[InteractionCompleteEvents] | None, default=None) — A list of events whose completion status needs to be checked. By default, only annotation_completed is checked
  • max_retries (int, default=60) — Maximum number of retries if calculations are not completed. Defaults to 60.
  • retry_interval_in_seconds (int, default=3) — Interval between retries in seconds. Defaults to 3.
  • return_topics (bool, default=True) — Whether to include the topic in the data.
  • return_annotation_data (bool, default=True) — Whether to include annotation info in the data.
  • return_builtin_props (bool, default=True) — Whether to include built-in properties in the data.
  • return_user_value_properties (bool, default=True) — Whether to include custom properties in the data.
  • return_custom_prompt (bool, default=True) — Whether to include Custom Prompt properties in the data.
  • return_similarities (bool, default=False) — Whether to include similarities in the data.
Returns
  • DataFrame | None — A DataFrame containing the retrieved data if calculations are completed, otherwise None.

get_dataset

def get_dataset(app_name: str, dataset_name: str) -> Dataset or None

Get a specific dataset by name.

Parameters
  • app_name (str) — Application name
  • dataset_name (str) — Dataset name
Returns
  • Dataset or None — The dataset if found, None otherwise

get_dataset_samples

def get_dataset_samples(app_name: str,
                        dataset_name: str,
                        offset: int = 0,
                        limit: int = 100) -> List[DatasetSample]

Get samples from a dataset with pagination.

Parameters
  • app_name (str) — Application name
  • dataset_name (str) — Dataset name
  • offset (int, default=0) — Number of samples to skip
  • limit (int, default=100) — Maximum number of samples to return
Returns

get_datasets

def get_datasets(app_name: str) -> List[Dataset]

Get all datasets for an application.

Parameters
  • app_name (str) — Application name
Returns

get_deployment

def get_deployment(app_name: str, deployment_name: str) -> Deployment or None

Get a specific deployment by name.

Parameters
  • app_name (str) — Application name
  • deployment_name (str) — Deployment name
Returns
  • Deployment or None — The deployment if found, None otherwise

get_deployments

def get_deployments(app_name: str) -> List[Deployment]

Get all deployments for an application.

Parameters
  • app_name (str) — Application name
Returns

get_interaction_by_user_interaction_id

def get_interaction_by_user_interaction_id(app_name: str,
                                           version_name: str,
                                           user_interaction_id: str) -> Interaction

Get a specific interaction by its user_interaction_id.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of application version
  • user_interaction_id (str) — Unique id of the interaction to get
Returns
  • Interaction — The interaction object, including the input, output, properties and other fields

get_interaction_type_config

def get_interaction_type_config(app_name: str,
                                interaction_type: str,
                                file_save_path: Union[str, None] = None) -> str

Write the auto-annotation YAML configuration file to the specified path.

Parameters
  • app_name (str) — Application name
  • interaction_type (str) — Interaction type name
  • file_save_path (str | None, default=None) — The path to save the configuration file to. If None, the file will not be saved.
Returns
  • str — The auto-annotation YAML configuration file as a string.

get_interaction_type_version_data

def get_interaction_type_version_data(app_name: str,
                                      version_name: str,
                                      interaction_type: str) -> InteractionTypeVersionData

Get interaction type version data.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Version name
  • interaction_type (str) — Interaction type name
Returns

get_interaction_types

def get_interaction_types(app_name: str) -> list[InteractionType]

Get the custom interaction types defined for the current application.

Parameters
  • app_name (str) — Application name
Returns

get_interactions_complete_status

def get_interactions_complete_status(app_name: str,
                                     version_name: str,
                                     user_interaction_ids: List[str],
                                     events_to_check: Optional[List[InteractionCompleteEvents]] = None) -> Dict[InteractionCompleteEvents, bool]

Get the completion status of interactions for specified events and user interaction IDs.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of application version
  • user_interaction_ids (List[str]) — A list of user interaction IDs to check for completion status.
  • events_to_check (List[InteractionCompleteEvents] | None, default=None) — A list of events whose completion status needs to be checked. By default, only annotation_completed is checked
Returns
  • Dict[InteractionCompleteEvents, bool] — A dictionary where keys are events from events_to_check and values are objects that indicating whether interactions for each event are completed and number of completed interactions {"annotation_completed": {"all_completed": True, "number_of_completed": 10}}

get_pentest_prompts

def get_pentest_prompts(app_name: str, probes: Optional[List[str]] = None) -> DataFrame

Get pentest prompts for the application.

Parameters
  • app_name (str) — Application name
  • probes (List[str] | None, default=None) — List of probes to get pentest prompts for. If None, all probes will be returned.
Returns
  • DataFrame — A pandas DataFrame containing the pentest prompts

get_raw_spans

def get_raw_spans(app_name: str,
                  version_name: str,
                  env_type: Union[EnvType, str, None] = None,
                  start_time: Union[datetime, int, None] = None,
                  end_time: Union[datetime, int, None] = None,
                  user_interaction_ids: Union[List[str], None] = None,
                  session_ids: Union[List[str], None] = None,
                  interaction_type: Union[str, None] = None,
                  include_incomplete: bool = False,
                  return_session_related: bool = True) -> list[dict]

Download raw OTEL spans matching the filter.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of application version
  • env_type (EnvType | str | None, default=None) — Environment type filter (PROD/EVAL).
  • start_time (datetime | int | None, default=None) — Filter spans whose interaction started at or after this time.
  • end_time (datetime | int | None, default=None) — Filter spans whose interaction started at or before this time.
  • user_interaction_ids (List[str] | None, default=None) — Restrict to interactions matching these user_interaction_ids.
  • session_ids (List[str] | None, default=None) — Restrict to interactions in these sessions.
  • interaction_type (str | None, default=None) — Restrict to a specific interaction type by name.
  • include_incomplete (bool, default=False) — Include interactions whose processing is not yet complete.
  • return_session_related (bool, default=True) — If True (default), expands the filter to include every interaction in the matched sessions, so incomplete / in-progress sessions are not truncated in the result. No-op when no user_interaction_ids / interaction_type filter is supplied (since the full version is already returned).
Returns
  • list[dict] — List of raw OTEL span dicts (the same shape originally uploaded via log_spans / log_spans_file).

get_user_value_properties

def get_user_value_properties(app_name: str,
                              interaction_type: Optional[str] = None) -> list[UserValuePropertyType]

Get the user value properties defined for the current application.

Parameters
  • app_name (str) — Application name
  • interaction_type (str | None, default=None) — Interaction type name
Returns

get_versions

def get_versions(app_name: str) -> List[ApplicationVersion]

Get all versions of an application as Version objects.

Note that "custom" attribute in ApplicationVersion is deprecated. Please use additional_fields instead.

Parameters
  • app_name (str) — Name of the application to get versions for
Returns

get_workspace_llm_default_model

def get_workspace_llm_default_model() -> LLMModelSettings

Get the workspace-level default LLM model used for property evaluation.

Returns
  • LLMModelSettings — The workspace-level LLM_PROPERTIES model that applies to every application unless overridden at the app level.

log_batch_interactions

def log_batch_interactions(app_name: str,
                           version_name: str,
                           env_type: Union[EnvType, str],
                           interactions: List[LogInteraction],
                           ignore_duplicates: bool = False) -> list of str

Log multiple interactions at once.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of the version to which this interaction will be added
  • env_type (EnvType | str) — Type of environment. Could be one of EnvType or 'prod', 'eval', 'pentest'
  • interactions (List[LogInteraction]) — The list of interaction data to log.
  • ignore_duplicates (bool, default=False) — If True, the system will ignore interactions with user_interaction_id that already exist in the system.
Returns
  • list of str — List of the uuids of the interactions

log_interaction

def log_interaction(app_name: str,
                    version_name: str,
                    env_type: Union[EnvType, str],
                    interaction: LogInteraction) -> str

Log a single interaction.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of the version to which this interaction will be added
  • env_type (EnvType | str) — Type of environment. Allowed values: 'eval', 'prod', 'pentest'
  • interaction (LogInteraction) — The interaction data to log.
Returns
  • str — The uuid of the interaction

log_spans

def log_spans(app_name: str,
              version_name: str,
              env_type: Union[EnvType, str],
              spans: Union[List[Span], List[dict]])

Send spans to the Deepchecks LLM backend.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of application version
  • env_type (EnvType | str) — Environment type (PROD/EVAL)
  • spans (List[Span] | List[dict]) — Either a list of Span objects, or a list of raw OTEL span dicts in the shape returned by get_raw_spans.

log_spans_file

def log_spans_file(app_name: str,
                   version_name: str,
                   env_type: Union[EnvType, str],
                   json_path: str)

Send spans to the Deepchecks LLM backend.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of application version
  • env_type (EnvType | str) — Environment type (PROD/EVAL)
  • json_path (str) — Path to the JSON file containing spans.

set_session_property_values

def set_session_property_values(app_name: str,
                                version_name: str,
                                session_id: str,
                                values: Dict[str, Any]) -> None

Set user value properties on a session.

Properties are auto-created as USER_VALUE kind if they don't already exist.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Version name
  • session_id (str) — The user-facing session ID
  • values (Dict[str, Any]) — Mapping of property display name to value. Numeric properties accept int/float, categorical accept str or list[str]. Pass None to clear a value.
Returns

update_app_basic_advanced_llm_models

def update_app_basic_advanced_llm_models(app_name: str,
                                         basic: Optional[str] = None,
                                         advanced: Optional[str] = None) -> None

Set the per-app Basic and/or Advanced LLM models.

Parameters
  • app_name (str) — Application name.
  • basic (str | None, default=None) — Model name for the Basic category. If omitted, the Basic model is left unchanged.
  • advanced (str | None, default=None) — Model name for the Advanced category. If omitted, the Advanced model is left unchanged.
Returns

update_app_default_llm_model

def update_app_default_llm_model(app_name: str, model_name: str) -> None

Set the per-app default LLM model used for property evaluation.

Parameters
  • app_name (str) — Application name.
  • model_name (str) — Name of the model to use for property evaluation in this application.
Returns

update_app_document_classification_config

def update_app_document_classification_config(app_name: str, enabled: bool) -> None

Enable or disable per-app document classification.

Parameters
  • app_name (str) — Application name.
  • enabled (bool) — Whether to enable document classification.
Returns

update_app_sampling_config

def update_app_sampling_config(app_name: str, prod_sampling_ratio: float) -> None

Update the per-app production sampling ratio.

Parameters
  • app_name (str) — Application name.
  • prod_sampling_ratio (float) — Production sampling ratio in the inclusive range [0, 1].
Returns

update_app_topics_config

def update_app_topics_config(app_name: str, enabled: bool) -> None

Enable or disable per-app topics generation.

Parameters
  • app_name (str) — Application name.
  • enabled (bool) — Whether to enable topics for this application.
Returns

update_app_translation_config

def update_app_translation_config(app_name: str,
                                  *,
                                  enabled: Optional[bool] = None,
                                  model_name: Optional[str] = None,
                                  with_text_splitting: Optional[bool] = None) -> None

Update the per-app translation configuration.

Parameters
  • app_name (str) — Application name.
  • enabled (bool | None, default=None) — Turn translation on or off for this application.
  • model_name (str | None, default=None) — Model name to use for translation.
  • with_text_splitting (bool | None, default=None) — Whether to enable text splitting for long inputs.
Returns

update_app_translation_detection_config

def update_app_translation_detection_config(app_name: str, model_name: str) -> None

Set the per-app translation language-detection model.

Parameters
  • app_name (str) — Application name.
  • model_name (str) — Model name to use for translation language detection.
Returns

update_application_version

def update_application_version(app_version_id: int,
                               name: Optional[str] = None,
                               description: Optional[str] = None,
                               additional_fields: Optional[dict] = None) ->     The response from the API

Create a new application.

Parameters
  • app_version_id (int) — Application version id to udate
  • name (str | None, default=None) — New version name
  • description (str | None, default=None) — New description of the version
  • additional_fields (dict | None, default=None) — Additional fields to add to the version
Returns
    • The response from the API*

update_batcher_config

def update_batcher_config(*,
                          batch_size: Optional[int] = None,
                          interval_seconds: Optional[int] = None,
                          should_wait_between_batches: Optional[bool] = None) -> None

Update the workspace batcher configuration.

Parameters
  • batch_size (int | None, default=None) — Number of interactions to send per batch.
  • interval_seconds (int | None, default=None) — How long the batcher waits before flushing a batch.
  • should_wait_between_batches (bool | None, default=None) — Whether the batcher inserts a wait between batches.
Returns

update_dataset

def update_dataset(app_name: str, dataset_name: str, new_dataset_name: str) -> Dataset

Update a dataset's name.

Parameters
  • app_name (str) — Application name
  • dataset_name (str) — Current dataset name
  • new_dataset_name (str) — New dataset name
Returns

update_deployment

def update_deployment(app_name: str,
                      deployment_name: str,
                      new_deployment_name: Optional[str] = None,
                      deployment_url: Optional[str] = None,
                      timeout: Optional[int] = None,
                      max_concurrent: Optional[int] = None,
                      max_retries: Optional[int] = None,
                      headers: Optional[List[Dict[str, Any]]] = None) -> Deployment

Update a deployment.

Parameters
  • app_name (str) — Application name
  • deployment_name (str) — Current deployment name
  • new_deployment_name (str | None, default=None) — New name for the deployment
  • deployment_url (str | None, default=None) — New URL for the deployment
  • timeout (int | None, default=None) — New timeout in seconds (5-300)
  • max_concurrent (int | None, default=None) — New max concurrent requests (1-20)
  • max_retries (int | None, default=None) — New max retries (0-5)
  • headers (List[Dict[str, Any]] | None, default=None) — New headers list
Returns

update_interaction

def update_interaction(app_name: str,
                       version_name: str,
                       user_interaction_id: str,
                       annotation: Optional[Union[AnnotationType, str]] = None,
                       annotation_reason: Optional[str] = None,
                       user_value_properties: Union[List[UserValueProperty], None] = None,
                       steps: Union[List[Step]] = None,
                       information_retrieval: Union[List[str], str] = None,
                       input: Union[str, None] = None,
                       output: Union[str, None] = None,
                       expected_output: Union[str, None] = None,
                       full_prompt: str = None,
                       history: Union[List[str], str] = None,
                       is_completed: Union[bool, None] = None,
                       started_at: Union[datetime, float] = None,
                       finished_at: Union[datetime, float] = None,
                       metadata: Optional[Dict[str, str]] = None,
                       tokens: Optional[int] = None,
                       input_tokens: Optional[int] = None,
                       output_tokens: Optional[int] = None) ->     None

Update a specific interaction by its user_interaction_id.

Parameters
  • app_name (str) — Application name
  • version_name (str) — Name of the version to which this interaction belongs
  • user_interaction_id (str) — Unique id of the interaction to update
  • annotation (AnnotationType | str | None, default=None) — Could be one of AnnotationType or 'good', 'bad', 'unknown'
  • annotation_reason (str | None, default=None) — String that explains the reason for the annotation
  • user_value_properties (List[UserValueProperty] | None, default=None) — List of user value properties for interaction
  • steps (List[Step], default=None) — List of steps taken during the interaction. New steps will be added to existing ones for previously incomplete interactions. Cannot be changed for completed interactions
  • information_retrieval (List[str] | str, default=None) — Information retrieval. New data will be added to existing ones for previously incomplete interactions. Cannot be changed for completed interactions
  • input (str | None, default=None) — Input data. Old value will be overridden by the new value. Note that once interactions are marked as completed, this value cannot be changed.
  • output (str | None, default=None) — Output data. Old value will be overridden by the new value. It cannot be modified for completed interactions.
  • expected_output (str | None, default=None) — Expected output data. Old value will be overridden by the new value. It cannot be modified for completed interactions.
  • full_prompt (str, default=None) — Full prompt data. Old value will be overridden by the new value. It cannot be modified for completed interactions.
  • history (List[str] | str, default=None) — History. New data will be added to existing ones for previously incomplete interactions. Cannot be changed for completed interactions
  • is_completed (bool | None, default=None) — Completed interactions will be shown in the system, and cannot be further modified except for annotations, topics and user value properties. Interaction completeness status will be not changed if None is sent
  • started_at (datetime | float, default=None) — Timestamp the interaction started_at at. Datetime format is deprecated, use timestamp instead
  • finished_at (datetime | float, default=None) — Timestamp the interaction finished at. Datetime format is deprecated, use timestamp instead
  • metadata (Dict[str, str] | None, default=None) — Metdata for the interaction.
  • tokens (int | None, default=None) — Token count for the interaction.
  • input_tokens (int | None, default=None) — Input token count for the interaction.
  • output_tokens (int | None, default=None) — Output token count for the interaction.
Returns
    • None*

update_interaction_type

def update_interaction_type(app_name: str,
                            interaction_type_id: int,
                            name: Optional[str] = None) -> None

Update interaction type for the current application.

Parameters
  • app_name (str) — Application name
  • interaction_type_id (int) — Interaction type id
  • name (str | None, default=None) — Name of the custom interaction type
Returns

update_interaction_type_config

def update_interaction_type_config(app_name: str,
                                   interaction_type: str,
                                   file = None) ->     None

Update the auto-annotation YAML configuration file for the interaction type within the application.

Parameters
  • app_name (str) — Application name
  • interaction_type (str) — Interaction type name
  • file — The path to the configuration file to update
Returns
    • None*

update_workspace_llm_default_model

def update_workspace_llm_default_model(model_name: str) -> None

Set the workspace-level default LLM model used for property evaluation.

Parameters
  • model_name (str) — Name of the model to use as the workspace default.
Returns

Updated 2 months ago

Did this page help you?

Updated 2 months ago

Did this page help you?