Annotation layers

A layer is one class of annotation output: object labels, hand pose, dense text, depth, and so on. Euler catalogs twelve, and eleven of them are servable today. Each one is a row that records what the layer produces, which models back it, which data kinds can carry it, how it is QA’d, which export formats carry it, and which knobs a team can set before a run.

The catalog is the single source of truth. The settings screen renders it, the run worker and the job queue consult the resolved per-project modes before spending on a lane, the coverage report reports against the same rows, and GET /v1/projects/{id}/annotation/settings serves it resolved for your project. Adding a layer is a catalog edit plus its lane, not a new code branch.

Two gates decide whether a layer runs. Applicability comes first and always wins: a layer whose data kind cannot carry it is reported as inapplicable whatever you set. The mode you choose in Pipeline settings decides the rest.

The catalog at a glance

Cost is the platform’s own per-episode estimate, used for the pre-run cost preview and the pre-flight budget guard. It is an estimate, not an invoice: the measured cost is metered on completion.

Layer	Id	Group	Servable	Runs as a job	Default mode	Est. cost / episode
Scene captions	`captions`	language	yes	no	in-run	$0.002
Object detection and masks	`object_labels`	perception	yes	no	on demand	$0.015
Semantic segmentation	`semantic_segmentation`	perception	yes	yes	off	$0.04
Monocular depth	`depth`	geometry	yes	yes	off	$0.03
Optical flow	`optical_flow`	geometry	yes	yes	off	$0.03
Hand pose	`hand_pose`	pose	yes	yes	in-run	$0.03
Full body pose	`body_pose`	pose	yes	yes	off	$0.035
Object tracks	`tracking`	perception	yes	yes	on demand	$0.02
Action segments	`subtasks`	language	yes	yes	on demand	$0.005
Dense description	`dense_text`	language	yes	yes	in-run	$0.02
Structured metadata	`metadata`	language	yes	yes	off	$0.003
3D representation	`threed`	geometry	no	yes	off	$0.02

“Default mode” is the catalog fallback. The mode actually in force for your project is resolved from the use case and target profile as well; see settings resolution.

One layer is catalogued and held back. The 3D representation lane is not deployed, so it is listed here and cannot be run. It is pinned off whatever your settings say, and enqueuing it returns 422 naming the reason. Eleven layers work today.

It stays in the catalog rather than being hidden because a shorter list would not tell you it exists. When the lane ships, it becomes runnable with no change to your settings.

Which data kinds carry which layer

A data kind is derived automatically from what an episode actually contains. Two layers are gated on a capability the data must physically have; the rest apply everywhere except the kinds listed against them.

Layer	Applies to	Does not apply to
`captions`	teleop, egocentric, drive-log, aerial-flight, vision-asset	state-only
`object_labels`	teleop, egocentric, drive-log, aerial-flight, vision-asset	state-only
`semantic_segmentation`	teleop, egocentric, drive-log, aerial-flight, vision-asset	state-only
`depth`	teleop, egocentric, drive-log, aerial-flight, vision-asset	state-only
`optical_flow`	teleop, egocentric, drive-log, aerial-flight	state-only, vision-asset
`hand_pose`	egocentric	every other kind (needs human hands in frame)
`body_pose`	egocentric	every other kind (same capability gate)
`tracking`	teleop, egocentric, drive-log, aerial-flight	state-only, vision-asset
`subtasks`	teleop, egocentric, drive-log, aerial-flight	state-only, vision-asset
`dense_text`	teleop, egocentric, drive-log, aerial-flight	vision-asset, state-only
`metadata`	every kind	—
`threed`	teleop, egocentric, drive-log, aerial-flight	state-only, vision-asset

Two layers also depend on another layer having run: tracking needs object_labels, and threed needs depth. The settings tile shows the dependency rather than failing at run time.

The layers one by one

Scene captions

One clear sentence per clip, plus object hints and failure notes, from the approved language lane.

Produces: a clip-level caption, suggested object names that seed the detection class list, and a note when the attempt looks unsuccessful.
QA: caption_agreement scores the caption against the frames.
Exports: jsonl, annotation_bundle, euler_certified_report.
Shows up in: the episode caption panel and search facets.
Configure: detail (brief, standard, rich), language, failure_notes, object_hints.
Feeds readiness: caption_coverage.

Object detection and masks

Grounded boxes and instance masks against your workspace vocabulary, with confidence routing and per-class quality floors.

Produces: open-vocabulary boxes, an instance mask confirming each box, and per-object properties if you ask for them.
QA: vocab_qa and score_floors. Low-confidence detections route to human review rather than being shipped.
Exports: detection_coco, jsonl, webdataset.
Shows up in: the episode overlay, the label workbench and search facets.
Configure: ontology_source (detect from the data, use the workspace ontology, or list them yourself), classes, attributes, confidence_floor (default 0.35), max_per_class (default 40), masks.

Semantic segmentation

Per-pixel class masks over the organized ontology, exported as indexed masks that training loaders read directly.

Produces: full-frame per-pixel class masks, optionally including background surfaces like floor, wall and table.
QA: boundary_depth_check and class_coverage. Boundaries are cross-checked against depth.
Exports: indexed_masks, jsonl.
Configure: ontology_source, classes, include_background, density (preview sample, keyframes, every frame), min_area.

Monocular depth

Per-frame relative depth, checked against the recording’s own depth stream where one exists.

Produces: relative depth maps, or depth rescaled onto the recording’s own depth stream when you pick aligned.
QA: source_depth_agreement and mask_boundary_check.
Exports: jsonl, depth_maps.
Configure: density, output (relative or aligned to source depth), cross_check, colormap.

Optical flow

Dense motion fields between frames, with forward-backward consistency as the built-in check.

Produces: per-pixel motion fields at the density you choose. Surfaces motion blur and dropped frames.
QA: fwd_bwd_consistency and occlusion_sanity. Forward and backward passes must agree.
Exports: jsonl, flow_fields.
Configure: density, max_side (working resolution, default 512), fwd_bwd_check.

Flow fields are large. Preview density keeps storage sane and still shows the motion profile.

Hand pose

Twenty one keypoints per hand, left and right, with handedness locked at the track level.

Produces: per-frame left and right hand keypoints, a rendered overlay video, and QA coverage numbers.
QA: pose_qa and pose_consistency.
Exports: coco_keypoints, annotation_bundle, jsonl.
Configure: show_confidence, min_confidence (default 0.3), smoothing (off, light, balanced, strong), handedness_lock, render_overlay.

handedness_lock decides handedness once per track, so a hand cannot flip sides mid clip. The lane’s adoption evidence, including the left/right integrity numbers, is recorded in benchmarks/t471.

Full body pose

Whole body keypoints for every person in frame, with the same tracking and overlay treatment as hands.

Produces: 17-point body skeletons, or the full 133-point whole-body set including hands, feet and face.
QA: pose_qa.
Exports: coco_keypoints, jsonl.
Configure: keypoint_set (coco17 or wholebody133), max_people (default 4), min_confidence, smoothing, render_overlay.

Object tracks

Objects followed across the clip with stable identities, from keyframe detection plus mask propagation.

Produces: tracks that hold identity through brief occlusions, editable in the timeline.
QA: track_continuity.
Exports: jsonl.
Depends on: object_labels.
Configure: classes (empty tracks whatever detection found), keyframe_interval_s (default 1.0), min_track_length, confidence_floor.

Track quality does not yet have a published benchmark; the measurement task is tracked as T-480.

Action segments

The clip split into labelled action spans, with a verb and a noun per span.

Produces: segment boundaries with consistent verb/noun labels, editable on the timeline.
QA: boundary_consistency.
Exports: jsonl, annotation_bundle.
Configure: taxonomy (verb and noun, or free text), verbs, granularity (coarse, standard, fine), min_span_s, language.

Boundary quality does not yet have a published benchmark; the measurement task is tracked as T-480.

Dense description

A moment-by-moment account of the whole clip, governed by the engagement’s written SOP.

Produces: long-form chronological description with a deterministic lint against the SOP checklist, and a per-clip QA receipt.
QA: sop_lint, self_consistency and caption_agreement.
Exports: annotation_bundle, jsonl.
Configure: sop_profile, style (chronological, narrative, technical), language, target_words (default 140), include_hands, custom_instructions.

Chronological is the safest style for training data. include_hands names the acting hand on episodes where hand pose ran.

Structured metadata

Filterable facts about every clip rather than free text.

Produces: scene type, environment, lighting, camera motion and task category as structured fields that become search facets and curation filters.
QA: spot_check_precision.
Exports: jsonl.
Configure: field_source (detect from the data, standard set, or list them yourself), fields, allow_new_values, language.

3D representation

Not servable yet. The point-cloud derivation lane is not deployed, so this layer cannot be run today. What follows is what it will produce.

Point clouds and camera path summaries derived from depth and the recording’s own spatial evidence.

Produces: point clouds derived from depth and camera intrinsics, plus a compact camera trajectory summary.
QA: source_depth_agreement.
Exports: jsonl, pointcloud.
Depends on: depth.
Configure: max_points (default 50000), frames, use_source_intrinsics, trajectory_summary.

A monocular-depth-derived cloud is categorically weaker evidence than multi-sensor SLAM fusion. Euler reports it as derived geometry, not as ground truth.

How a layer is QA’d

Every layer carries QA check ids in the catalog, and those ids are what appear as evidence chips on the episode and in the coverage report. They are identifiers, not formulas: the report shows what was checked and what it found, and a check that could not run reports not_assessable with the reason rather than a made-up number.

Read the result with:

curl -fsS -H "Authorization: Bearer $EULER_TOKEN" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/annotation/coverage"

For every layer that returns: how many episodes carry it, how many could, the quality signal the layer itself measured, the QA check ids behind it, and the last run that produced it. Narrow with ?layer= or ?kind=.

Models behind the layers

Layers resolve their model through the registry (GET /v1/models), which records the active model per surface, its adoption evidence, its cost note, and the vetted alternatives you may switch to. Perception surfaces run self-hosted models only: customer frames never leave the platform boundary.

Model selection is license-gated as well as accuracy-gated, and the registry records why a higher-scoring model was not adopted. Four examples, quoted from the registry rows:

Hand pose runs rtmdet-nano-hand+rtmpose-m-hand5, Apache-2.0 end to end. The higher-scoring mesh models (WiLoR, HaMeR) are CC-BY-NC or MANO research-only and are legally unusable on commercial engagements.
Body pose runs rtmlib/wholebody, Apache-2.0 end to end. Meta’s Sapiens scores higher and is CC-BY-NC, so it is not available for commercial work.
Depth runs depth-anything/Depth-Anything-V2-Small-hf, which is Apache-2.0. The Base, Large and Giant checkpoints of the same family are CC-BY-NC and are blocked for commercial use, as is Metric3D.
Semantic segmentation runs tue-mps/coco_panoptic_eomt_large_640. Both it and the Mask2Former checkpoints are MIT and clear the gate; OneFormer’s license was unclear in the research pass, so it is not offered as a default.

The registry names what actually runs, not what we would prefer to run. Optical flow serves torchvision/raft_large. SEA-RAFT scores higher and carries the same permissive license, but publishes no redistributable checkpoint, so the deployed lane serves RAFT and reports that it did. A registry that reports an aspiration is worse than no registry.

The full mapping, including the benchmark reference behind each adoption, is served live:

curl -fsS -H "Authorization: Bearer $EULER_TOKEN" "$EULER_BASE_URL/v1/models"

How a layer is briefed

Every layer backed by a language or vision model is briefed before it runs, and the briefing is specific rather than generic: it carries what this class of data physically is, what your workspace exists to do, what the output is ultimately being trained on, and your own vocabulary.

Every briefing also carries the same guardrails. Describe only what is present. Never invent objects, people, counts, times, distances, measurements or outcomes. When you cannot tell, say “not assessable” and why. Do not carry anything over from another recording.

That is the whole reason a Euler annotation is worth reviewing: a missing annotation is recoverable, a confidently invented one is not, because you cannot tell it apart from a correct one.

Turn layers on, configure them, and see the cost before you spend: Pipeline settings.
Run a heavy layer after the fact: Annotation jobs.
See how good the output actually is: Automated review pass.
Work through a project recording by recording: Review workbench.
Understand what the readiness score measures: Readiness.