Children Annotation Aggregation

What Is It?

Children Annotation Aggregation automatically annotates parent interactions based on their children's annotations. This is designed for hierarchical interaction types like Agent, Chain, and Root, where a parent's quality depends on its child interactions.

Why Use It?

In agentic LLM systems, interactions are hierarchical:

An Agent spawns multiple Tool and LLM calls
A Chain orchestrates a sequence of child steps
A Root represents the top-level entry with nested children

If a child interaction fails (annotated as "bad"), the parent should reflect this. Children annotation aggregation automates this by propagating annotations upward based on configurable rules.

YAML Configuration

There is a children block in your default auto-annotation pipeline:

# Children-based annotation: This block labels samples as "bad" if more than 20% of their children are labeled as "bad".
  - type: children
    annotation: bad
    conditions:
      - mode: simple
        operator: GT
        children_annotation: bad
        value: 0 # Threshold: should be between 0 and 1, representing the fraction of children with the specified annotation.

Configuration Fields

Field	Values	Description
`type`	`children`	Identifies this as a children annotation block
`annotation`	`good` / `bad` / `unknown`	Annotation to apply if conditions match
`relation_between_conditions`	`OR` / `AND`	How to combine multiple conditions (default: `OR`)

Condition Fields

Field	Values	Description
`mode`	`simple`	Evaluation mode
`operator`	`GT` / `GE` / `LT` / `LE`	Comparison operator
`children_annotation`	`good` / `bad` / `unknown`	Annotation to look for in children
`value`	0-1	Threshold (fraction of children)
`interaction_types`	list or null	Filter to specific child types (optional)

Common Patterns

Mark bad if ANY child is bad (default behavior):


* type: children
    annotation: bad
    conditions:
      - mode: simple
        operator: GT
        children_annotation: bad
        value: 0

Mark bad if more than half of children are bad:

* type: children
    annotation: bad
    conditions:
      - mode: simple
        operator: GT
        children_annotation: bad
        value: 0.5

Only consider specific child types:

* type: children
    annotation: bad
    conditions:
      - mode: simple
        operator: GT
        children_annotation: bad
        value: 0
        interaction_types: ["tool", "llm"]