Data Upload
The different functions and knowhow needed to upload data into the Deepchecks platform.
The data hierarchy in Deepchecks starts with applications, each representing a use-case. Each application contains different versions, and the data in each version can either be part of the Evaluation set (which should be common to all versions) or the Production set. Therefore, when uploading data, we need to specify the location to which we want the data to be uploaded.
dc_client.log_batch_interactions(
app_name="DemoApp", version_name="v1", env_type=EnvType.EVAL,...
Before any data can be uploaded, an application with the appropriate name must first exist.
dc_client.create_application('MyApp', app_type=ApplicationType.QA)
Batch Upload
Used either for uploading evaluation set data from the research (or staging) environment or as part of a periodic job to upload the collected production data.
Deepchecks represent data samples as LogInteraction. Prior to batch data uploading, we need to organize our data as LogInteraction objects. In the example below, the data is first converted into a list of LogInteraction objects and then uploaded to the evaluation set of version 'v1'.
interactions = [LogInteraction(input=row["input"],
output=row["output"],
user_interaction_id=row["id"],
interaction_type="Q&A",
session_id="session-1") for _, row in df.iterrows()]
dc_client.log_batch_interactions(
app_name="DemoApp", version_name="v1", env_type=EnvType.EVAL, interactions=interactions)
Stream Upload
Deepchecks' fractional logging feature is primarily used in inference server integration. It enables tracking of an interaction as it progresses through the inference pipeline by utilizing the is_completed flag. The interaction's properties remain uncalculated and it stays hidden from the user interface until marked as completed.
dc_client.log_interaction(
app_name="DemoApp",
version_name="v1",
env_type=EnvType.EVAL,
user_interaction_id="id-1",
history=["Hi", "Hello! how can i assist you today?"],
input="My Input",
user_value_properties=[UserValueProperty("User Region", "US")],
is_completed=False,
interaction_type="Q&A",
session_id="session-1",
)
...
dc_client.update_interaction(
app_name="DemoApp",
version_name="v1",
user_interaction_id="id-1",
information_retrieval=["First doc", "Second doc"],
user_value_properties=[UserValueProperty("# of Documents", 2)]
)
...
dc_client.update_interaction(
app_name="DemoApp",
version_name="v1",
user_interaction_id="id-1",
output="My Output",
is_completed=True
)
Multi-Step Application
For applications that contain more than a single logical step, we will want to log each step that was executed. In Deepchecks, these intermediate steps are logged under the steps argument in an interaction.
from deepchecks_llm_client.data_types import Step
intermediate_steps = [
Step(
name="Router",
value="Go to Question Answering agent"
),
Step(
name="PII_Removal",
value="anonymized user input",
),
Step(
name="Question Answering",
value="model answer 1",
),
]
interaction_w_steps = LogInteraction(
user_interaction_id="id-1",
input="user input 1",
output="model answer 1",
steps=intermediate_steps,
interaction_type="Q&A",
)
Steps can also be updated via update_interaction endpoint:
dc_client.log_interaction(
app_name="DemoApp",
version_name="v1",
env_type=EnvType.EVAL,
user_interaction_id="id-1",
input="user input 1",
steps=[Step(name="Router",
value="Go to Question Answering agent")],
is_completed=False,
interaction_type="Q&A",
)
...
dc_client.update_interaction(
app_name="DemoApp",
version_name="v1",
user_interaction_id="id-1",
steps=[Step(name="PII_Removal",
value="anonymized user input")],
)
...
dc_client.update_interaction(
app_name="DemoApp",
version_name="v1",
user_interaction_id="id-1",
steps=[Step(name="Question Answering",
value="model answer 1")],
output="model answer 1",
is_completed=True
)
User Values Properties and Timestamps
User Value Properties are your way of supplying additional information about the sample to the Deepchecks platform. User Value properties are usually either metrics that are designed to be part of the Automatic Annotations or metadata used both for logging purposes as well as for Root Cause Analysis (RCA). You may provide a textual reason for a given property value.
Timestamps are provided via the started_at and finished_at parameters. They are used to calculate the application's latency and to mark the time the samples were processed for production monitoring.
from deepchecks_llm_client.data_types import LogInteraction
interaction = LogInteraction(user_interaction_id="id-1",
input="user input 1",
output="model answer 1",
started_at='2024-09-01T23:59:59',
finished_at=datetime.now().astimezone(),
interaction_type="Q&A",
user_value_properties=[
UserValueProperty("My Numeric Property", 1.5),
UserValueProperty("My Categorical Property", "Low quality",
reason="Bad grammer")]
)
Updating Existing Interactions
In certain instances, auxiliary information may arrive after the interaction has already been logged (as completed) on the Deepchecks platform. Common examples include annotations that can be obtained later through direct user feedback or an internal quality assurance agent, as well as User Value Property values.
Deepchecks do not permit updates to the data fields on completed interactions, as these changes can impact the property calculation process. If such modifications are necessary, you can delete the interaction and then re-upload it.
dc_client.update_interaction(
app_name="DemoApp",
version_name="v1",
user_interaction_id="id-1",
annotation="bad",
annotation_reason=None,
user_value_properties=[UserValueProperty("My User Value Property", 1.5)]
)
Uploading data directly from the UI
Notice - you can also upload data to the system using CSV/JSONL format directly from the UI
Updated about 1 month ago