Annotation layers

A layer is one class of annotation output: captions, object outlines, hand pose, depth, a 3D scene, and so on. Euler catalogs thirteen, all of them servable. Each one records what it produces, which data kinds can carry it, how it is checked, which export formats carry it, which settings a team can change, and where the review screen shows it.

The catalog is the single source of truth. Pipeline settings renders it, the run and the job queue consult your project’s resolved modes before spending, and GET /v1/projects/{id}/annotation/settings serves it resolved for your project.

The catalog at a glance

“Default mode” is the catalog fallback. The mode in force for your project is resolved from your use case and target profile as well; see settings resolution. Costs are shown per layer in Pipeline settings before you run.

Which data kinds carry which layer

A data kind is derived from what an episode actually contains. Three layers need a capability the data must physically have; the rest apply everywhere except the kinds listed against them.

Some layers build on another: tracking needs object_labels, dense_text needs subtasks, and threed needs depth. The settings tile shows the dependency rather than failing at run time.

The layers one by one

Scene captions

One clear sentence per clip, plus object hints and failure notes.

• Produces: a clip-level caption, suggested object names that seed the detection class list, and a note when the attempt looks unsuccessful.
• Checked by: caption agreement with the frames.
• Exports: jsonl, annotation_bundle, the certified report.
• Settings: guidelines, detail (brief, standard, rich), language, failure_notes, object_hints.

Object detection and masks

Object outlines as polygons from instance masks, with their boxes, against your workspace vocabulary, routed by confidence and per-class quality floors.

• Produces: named outlines and boxes, and per-object properties if you ask for them.
• Checked by: vocabulary checks and score floors. Low-confidence detections route to review rather than being shipped.
• Exports: detection_coco, jsonl, webdataset, and the annotation bundle (with polygons).
• Settings: ontology_source, classes, attributes, guidelines, perspective, occlusion and part rules, confidence_floor, max_per_class, annotation_shape (outlines, the default, or boxes only), frames_per_clip.

Semantic segmentation

Per-pixel class masks over your organised ontology, exported as indexed masks.

• Checked by: boundary checks against depth, and class coverage.
• Exports: indexed_masks, jsonl.
• Settings: ontology_source, classes, include_background, density, min_area.

Monocular depth

Depth in metres for every pixel of each sampled keyframe, with the camera’s intrinsics estimated.

• Checked by: agreement with the recording’s own depth where one exists, and the estimated focal length against the camera’s known lens.
• Exports: depth.npz, 16-bit PNGs with their scale, and a JSON that says whether the values are metric or relative.
• Settings: density: preview (16 keyframes per episode), keyframes (48) or full (300).

Optical flow

Dense motion between frames, with forward-backward consistency as the built-in check. Surfaces motion blur and dropped frames.

• Settings: density, max_side, fwd_bwd_check.

Hand pose

Twenty one keypoints per hand, left and right, never swapped: handedness is decided once per track.

• Produces: a full-density landmark file per episode, and coverage and gap numbers. On first-person data only the camera wearer’s hands are kept.
• Checked by: pose QA and pose consistency.
• Exports: coco_keypoints, annotation_bundle, jsonl.
• Settings: guidelines, perspective, show_confidence, min_confidence, smoothing, handedness_lock, an automated review check.

The review screen paints the skeletons on the clip live; no overlay video is rendered or stored.

Full body pose

Whole-body keypoints for every person in frame.

• Exports: coco_keypoints, jsonl.
• Settings: keypoint_set (17 body points, or the 133-point whole-body set), max_people, min_confidence, smoothing.

Object tracks

Each object followed through the clip with its outline on every tracked frame (two a second), consolidated so one object is one track.

• Checked by: track continuity.
• Settings: classes, method, keyframe_interval_s, min_track_length, confidence_floor.
• Clean up object tracks in Pipeline settings re-decides identities from the tracker’s raw output without a GPU.

Action segments

The clip split into named steps, a verb and a noun per span.

• Settings: taxonomy, verbs, granularity, min_span_s, language.

Event tagging

Timestamped point events in your own schema, for third-person video, with the evidence behind each event and a reviewer’s editor on the review screen.

Dense actions

Every action inside each step, second by second: “put down phone” inside “arranging boxes on the desk”, with the acting hand.

• Checked by: each action sits inside its step, and agrees with the caption.
• Settings: temporal_detail, taxonomy, min_action_s, max_actions_per_segment, include_hands, language, custom_instructions.

Structured metadata

Scene type, environment, lighting, camera motion and task category as filterable fields that become search facets and curation filters.

• Settings: field_source, fields, allow_new_values, language.

3D representation

The camera’s path through the whole episode, point clouds of the scene, and experimental 3D hands, in one metric, gravity-aligned frame.

• Produces: the camera trajectory (about ten poses a second, with a confidence per pose), a fused point cloud and one cloud per depth keyframe, object centres in 3D, and hands lifted from the 2D hand pose through the depth.
• Honest numbers: the path length is reported smoothed to about once a second, because head bob and tracker jitter inflate a raw path several times over. Both values are kept.
• Exports: the combined 3D file, per-keyframe clouds, the trajectory table and its summary, and the 3D hands (experimental, and easy to leave out).
• Settings: max_points, trajectory_pose_hz, trajectory_window_s.
• Depends on: depth. Rebuild 3D from stored depth in Pipeline settings re-derives it, in colour, without a GPU.

3D hands are a lift of 2D keypoints through estimated depth and are marked experimental: they exist near depth keyframes, not on every frame, and are not yet at training-grade accuracy. See Limits.

How a layer is checked

Every layer carries named checks, and those are what appear as evidence on the episode and in the coverage report. They are identifiers, not formulas: the report shows what was checked and what it found, and a check that could not run reports not_assessable with the reason rather than a made-up number.

curl -fsS -H "Authorization: Bearer $EULER_TOKEN" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/annotation/coverage"

For every layer: how many episodes carry it, how many could, the quality signal the layer measured, the checks behind it, and the last run that produced it. Narrow with ?layer= or ?kind=.

How models are chosen

Every layer resolves its model through one registry, shown in Settings under Tenant, Model defaults; a project can choose its own where a choice is offered (see Models per project). Two rules govern it. No unmeasured swaps: a model is adopted only on evidence from our own benchmark harness. Licence first: a model whose licence does not allow commercial use, or that depends on an asset that does not, is never used on customer work, however well it scores. Perception runs on self-hosted models, so your frames never go to an external labelling service.

How a layer is briefed

Every layer backed by a language or vision model is briefed before it runs with what this class of data physically is, what your workspace exists to do, what the output will train, and your own vocabulary. Every briefing carries the same guardrails: describe only what is present; never invent objects, people, counts, times, distances or outcomes; when unsure, say “not assessable” and why. A missing annotation is recoverable, a confidently invented one is not.

Next

• Turn layers on and see the cost: Pipeline settings.
• Run a layer after the fact: Annotation jobs.
• How good the output is: Automated review pass.
• Look at it: Review.