Automated review pass
Coverage tells you a layer produced something. Quality tells you what the layer measured about itself. Neither answers the question a reviewer actually asks: is this annotation any good? A caption can exist and be wrong. A step list can cover the whole timeline and describe the wrong steps.
The review pass answers that. It looks at a sample of frames from a recording together with what each layer produced, and records a graded verdict per layer, with the evidence it looked at and what it would improve.
Running it
It runs like any other layer job, under the id annotation_review:
curl -fsS -X POST -H "Authorization: Bearer $EULER_TOKEN" \
-H "Content-Type: application/json" \
"$EULER_BASE_URL/v1/projects/$PROJECT/annotation/jobs" \
-d '{"layer_id": "annotation_review"}'Scope it to specific recordings with episode_ids, watch it on the same job
stream, and cancel it the same way. See annotation jobs.
It is not one of the annotation layers. It produces no annotation, it grades the ones that exist, so it has no tile in pipeline settings and no mode. That is on purpose: a layer that could be switched to run inside every pipeline run is a layer you could pay for twice without deciding to.
What a verdict contains
Per layer, per recording:
| Field | What it is |
|---|---|
| Score | 0 to 1, higher is better |
| Not assessable | True when nothing could be judged. The score is then empty |
| Reason | Why the score is what it is, or why nothing could be judged |
| Evidence | What the reviewer actually looked at |
| Improvements | What it would change about the output |
| Recommended prompt and config | A proposal for how the layer should be briefed or configured next time |
| Model and provider | Which reviewer produced the verdict |
| Reviewed at | When |
A score and “not assessable” can never both be present. That is what stops an invented number from reading like a measured one.
Which layers it can judge
Scene captions, object detection, object tracks, action segments, dense description, structured metadata, hand pose, body pose and semantic segmentation.
Depth, optical flow and 3D are deliberately excluded. Those lanes ship their own measured cross-checks, and a vision model looking at a handful of sampled frames cannot judge a depth map better than that cross-check already does. They keep their own numbers rather than being given an opinion.
The rules that make it worth trusting
- A verdict is only kept for a layer that actually produced output on that recording. Nothing is graded in absentia.
- A reply without a usable score is recorded as not assessable, with the reason. It is never rounded to something, and never dropped silently.
- Recommendations are recorded, never applied. A proposed better briefing or configuration lands on the record as a proposal. A human decides whether to take it. A model rewriting the instructions given to a model, with nobody in between, produces annotation whose provenance nobody can explain.
- It degrades honestly. With no reviewer model configured, the job fails with a plain reason. It never writes a verdict it did not earn.
- Your frames stay inside the platform boundary, on the same approved vision lanes every other layer uses.
Where the verdicts show up
On the review workbench, as evidence beside the layer they grade.
They also fill a real hole in the coverage view. A layer with no self-check of its own used to read as permanently unassessable; once a review has looked at it, the coverage view reports the reviewer’s score for it instead. A layer’s own measurement always wins over the reviewer’s opinion where both exist.
Cost
The review pass is one small multimodal call per recording, over a handful of frames, so it is one of the cheapest things you can run. Like every layer, its estimate is checked against your budget before anything starts, and what is metered afterwards is the measured cost.
Next
- The queue this runs on: Annotation jobs.
- What each layer produces: Annotation layers.
- What the platform measures about your data overall: Data utility.