Data Component for Prompt Properties

Overview

The Data component in the prompt property creation and edit flow provides full customizability over which data fields your property uses for evaluation. This component allows you to precisely control what information is available to the LLM judge when calculating the property score, ensuring your property has access to exactly the data it needs for accurate evaluation.

The Data component operates on two levels:

Current Span Data Fields - Select which data fields from the current span (interaction) will be used
Descendant Scopes - For hierarchical use-cases, access data from child spans at different levels

Selecting Current Span Data Fields

When configuring a prompt property, you first select which data fields from the current span will be provided to the LLM for evaluation. Available data fields include:

Input - The user's input or query
Output - The generated response
Full Prompt - The complete prompt used for generation
Information Retrieval - Retrieved context or knowledge base information
History - Previous conversation context
Expected Output - The expected or reference output
Custom Fields (steps) - Any custom data fields defined for your application

You can select any combination of these fields based on what your property needs to evaluate. For example, if you're creating a property that only evaluates the quality of the output, you might choose to include only the "Output" field.

Must vs Optional Fields

Each "current span" data field you select can be marked as either "Must" or "Optional":

Must Fields

If a "Must" field is missing in a specific interaction, the property cannot be calculated
The property score will return N/A with an explanation indicating which required field was missing
Use "Must" for fields that are essential to your property's evaluation logic

Example: If your property evaluates "Response Completeness" and requires both the input and output to determine if the response fully addresses the query, you would mark both "Input" and "Output" as "Must".

Optional Fields

If an "Optional" field is missing, the property will still be calculated using the available data
The property will ignore the missing optional field and proceed with evaluation
Use "Optional" for fields that provide additional context but aren't strictly necessary

Example: If your property evaluates "Response Quality" and benefits from having the full prompt context but can still function with just input and output, you would mark "Full Prompt" as "Optional".

Descendant Scopes for Hierarchical Use-Cases

For complex hierarchical use-cases, such as agentic applications, prompt properties can access not just the current span's data fields but also data from descendant (child) spans. This capability is essential for evaluating multi-step interactions where the property needs to consider information from nested operations.

Adding a Scope

To access descendant data, you add a "Scope" to your property configuration. Each scope defines:

Depth Level - How many levels down from the current span you want to look
- 1 level down - Direct children only
- 2 levels down - Children and grandchildren
- All levels - All descendants
Interaction Types - Which types of child spans to include
- You can filter to specific interaction types (e.g., only "Tool" spans, only "LLM" spans)
- Or include all interaction types at that level
Data Fields - Which data fields from those descendant spans to include
- You can select any combination of fields (Input, Output, Full Prompt, etc.)
- All descendant data fields are "Optional"

Scope Examples

Example 1: All Descendant Tool Spans

Use Case: Evaluate whether all tool calls in an agentic interaction were appropriate.

Scope Configuration:

Depth: All levels
Interaction Types: Tool spans only
Data Fields: Input, Output

This scope will collect data from all Tool spans at any level below the current span, allowing the property to evaluate the entire tool usage pattern.

Example 2: Direct Children LLM Spans

Use Case: Evaluate the quality of responses from an agent's direct LLM children.

Scope Configuration:

Depth: 1 level down (direct children only)
Interaction Types: LLM spans only
Data Fields: Output, Full Prompt

This scope will only look at LLM spans that are direct children of the current span, ignoring deeper levels and other span types.

Example 3: Multi-Level Analysis

Use Case: Evaluate a complex agentic workflow that involves multiple levels of tool calls and LLM responses.

Scope Configuration:

Scope 1:
- Depth: 1 level down
- Interaction Types: LLM spans
- Data Fields: Output
Scope 2:
- Depth: All levels
- Interaction Types: Tool spans
- Data Fields: Input, Output

This configuration uses multiple scopes to gather different types of information from different parts of the hierarchy, providing comprehensive context for evaluation.

Flexibility and Granularity

The Data component's two-level approach (current span fields + descendant scopes) provides:

Full Flexibility - Choose exactly which data fields are available to your property
Granular Control - Specify precise depth levels and interaction type filters
Token Efficiency - Only send the data fields you actually need to the LLM, reducing token usage and costs while maintaining evaluation accuracy
Tailored Properties - Build properties that perfectly match your specific use case requirements

This granularity enables you to create highly specialized properties that can handle everything from simple single-span evaluations to complex multi-level agentic workflows, ensuring your evaluation criteria are precisely aligned with your application's architecture and requirements.

Best Practices

Start Simple - Begin with just the current span data fields you need, then add scopes if hierarchical access is required
Use "Must" Sparingly - Only mark fields as "Must" if they are truly essential. Overusing "Must" can result in N/A scores
Consider Performance - Accessing many descendant levels or including many fields can increase evaluation time and cost
Match Your Architecture - Design your data selection to match your application's span hierarchy and data flow
Test Thoroughly - Use the "test" feature to validate that your data configuration works correctly with your actual interactions