Curation

Curation is how you turn everything you have into the subset you should actually train on. Euler models it as a plan: an ordered sequence of strategies, each one a row in a registry, executed against your project’s episodes and their signals, producing a subset plus a report that attributes every dropped episode to the stage that dropped it.

Nothing disappears from a curated subset without a stage owning the decision.

Strategies

Every curation move is a registry row with a stable id, a plain-language statement of what it optimizes, when to reach for it, what it trades away, typed parameters with defaults, and the signals it needs. Seven ship today.

There are three kinds of stage. A filter removes or down-weights episodes. A weight changes preference without removing anything. A select is what actually cuts to a target size, which is why diversity_cover is the last stage in almost every plan.

Readiness floor

Hold everything to a readiness bar, or exclude a specific fault you never want to train on. When: whenever a readiness pass has run. Use the per-check list when one fault matters more than the overall score, for example never including episodes flagged for erratic control. Trade-off: a hard floor can empty a small dataset; down-weighting keeps weak episodes available as filler when nothing better exists.

Parameters: min_readiness (default 0.5), exclude_flags, component_floors, mode (drop or down-weight), down_weight (default 0.25), unscored (default keep).

Annotation complete

Require the annotation layers your target profile consumes. When: the project has a declared target profile, or you are exporting for a trainer that will crash on a missing layer. Trade-off: strict on a partially annotated project this can drop most of the data. Run the missing layer first, or down-weight instead of dropping.

Parameters: required_layers (empty derives them from the target profile), mode, down_weight (default 0.3).

Remove near duplicates

Drop near-identical recordings. When: almost always, and especially on fleet data where the same station or route was recorded over and over. Skip it when duplicates are the point, for example when you are measuring repeatability. Trade-off: genuinely repeated attempts at a hard task look like duplicates; raise the threshold if you want to keep repeats.

Parameters: similarity_threshold (default 0.97), keep (default highest_readiness).

Prefer the hard cases

Weight toward recordings where label confidence was low, where the caption and the video disagreed, or that sit far from everything else the models have seen. When: once the obvious data is already in the training set and the model has stopped improving. Also useful for building a review queue. Trade-off: hard often means messy. Pair it with a readiness floor so you get genuinely hard data rather than broken recordings.

Parameters: strength (default 1.0), prefer (default hard), drop_fraction (default 0.0).

Focus on real failures

Pull the clusters your model feedback marked as real-world failures into the subset instead of averaging them away. When: after a deployment produced failure reports, or after the feedback loop clustered them. This is the strategy that turns a failure into next week’s training data. Trade-off: oversampling failures skews the distribution. Keep the boost modest unless you are deliberately building a corrective set.

Parameters: boost (default 2.0), guarantee (default false), clusters.

Balance across a facet

Impose a quota per task, data kind, label class, metadata field or cluster, so the most-recorded mode does not dominate what the model sees. When: one task, site or class makes up most of your data. Also the honest way to build an evaluation set. Trade-off: capping the majority throws away real data. If the production distribution genuinely is lopsided, balancing moves you away from it.

Parameters: facet (default task), max_share (default 0.4), min_per_value.

The strategies endpoint reports which facets your project actually has, how many distinct values each carries, and the largest share, so you can pick a facet that means something.

Cover the whole dataset

Greedy selection where each pick is the episode that adds the most territory the subset does not already have. When: whenever you are cutting a large dataset down to a training or evaluation budget. Trade-off: maximum variety also over-represents the rare and the strange relative to production; blend in preference to pull it back toward quality.

Parameters: target_size, preference_weight (default 0.5), method (default facility_location), max_full_matrix_episodes (default 4000).

Above the episode ceiling, facility-location downgrades to greedy k-center, which needs far less memory. The downgrade is reported rather than done silently.

What strategies can run here

Before composing anything, ask the project what it can support:

curl -fsS -H "Authorization: Bearer $EULER_TOKEN" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/curation/strategies"

The response gives the registry, the pool size, the available facets, and a signals block reporting which signals actually exist: embeddings and which embedding space, readiness scores, readiness components, facets, difficulty, feedback, annotation layers. A strategy whose signals are missing comes back with applicable: false and a plain-language reason, so the screen greys it out instead of failing after the fact.

Composing a plan

A plan is a name, an optional target_size, a seed, and an ordered list of stages. Each stage names a strategy and optionally overrides its parameters.

{
  "name": "Q3 manipulation train set",
  "target_size": 500,
  "seed": 7,
  "stages": [
    {"strategy": "readiness_floor",
     "params": {"min_readiness": 0.6, "exclude_flags": ["erratic_control"]}},
    {"strategy": "dedup", "params": {"similarity_threshold": 0.98}},
    {"strategy": "annotation_complete"},
    {"strategy": "failure_focus", "params": {"boost": 1.5}, "weight": 1.0},
    {"strategy": "balance", "params": {"facet": "task", "max_share": 0.35}},
    {"strategy": "diversity_cover", "params": {"preference_weight": 0.5}}
  ]
}

Order is meaningful and visible. Dropping duplicates before balancing gives a different subset than the reverse, and the report shows both stages’ decisions. A stage can be disabled in place with "enabled": false.

Execution rules, all chosen so a result can be defended:

• Stages run in the order given.
• Every episode that leaves the subset is attributed to exactly one stage. If the plan still overshoots its target after the last stage, an explicit final trim stage is recorded rather than a list being silently sliced.
• A stage that cannot run (no embeddings, no feedback, only one task) is skipped and says so. It never fails the plan and never silently no-ops.
• Results are deterministic given the seed. Every ordering ends in the episode id, and the seed is consulted only where a choice is genuinely arbitrary.
• An unknown parameter key raises. A typo must not silently curate a different dataset than the one you asked for.

Preview, then apply

Preview runs the plan and returns the report without saving anything:

curl -fsS -X POST -H "Authorization: Bearer $EULER_TOKEN" \
  -H "Content-Type: application/json" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/curation/preview" \
  -d '{"plan": {"name": "Preview", "target_size": 500, "stages": [{"strategy": "dedup"}, {"strategy": "diversity_cover"}]}}'

You get the executed plan, the report, the selected episode ids (capped by max_episode_ids, with selected_truncated telling you when the list was cut), the pool size and the signal availability.

Apply runs the same plan and freezes the result as a saved slice, so it flows through every surface slices already reach, including export:

curl -fsS -X POST -H "Authorization: Bearer $EULER_TOKEN" \
  -H "Content-Type: application/json" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/curation/apply" \
  -d '{"plan": {...}, "name": "Q3 manipulation train set", "purpose": "train"}'

Both endpoints accept an optional feedback_records batch in the same shape the feedback endpoints take; supplying it links episodes to real failure clusters so failure_focus has something to act on. Reads need viewer; apply needs engineer and is audited.

Let Euler propose the plan

curl -fsS -H "Authorization: Bearer $EULER_TOKEN" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/curation/recommend"

Returns a plan built from what this project actually has, plus reasons explaining each stage that was included and skipped explaining each one that was not, the signal availability, and the expected maximum subset size. Preview it, edit it, then apply it.

The older curate endpoint

POST /v1/projects/{id}/curate is the earlier single-move endpoint: greedy facility-location over the visual vectors blended with a per-episode utility score, restricted to accepted episodes, frozen as a train slice. It still works and is what the copilot’s curate_subset tool calls.

The curation plan endpoints supersede it. New integrations should use curation/preview and curation/apply, which express the same selection as a diversity_cover stage and can compose it with the other six strategies.

Next

• Feed curated slices into a dataset version: Agents and exports.
• Ask the copilot to do it conversationally: Euler Copilot.