Skip to content
Euler Docs

Annotation jobs

Heavy annotation layers run as asynchronous jobs. The request enqueues and returns immediately with 202 and a job record; you then poll the job or subscribe to a Server-Sent Events stream for live progress.

Why heavy lanes are jobs

The platform’s edge proxy caps a single request at 300 seconds. That budget is not enough for the GPU lanes, and the failure mode was actively misleading: a 291-second hand-pose call returned 502 to the browser and then completed server-side about 25 minutes later. The customer saw a failure for work that had actually succeeded. A cold GPU start alone costs roughly 200 seconds before any real work begins, and the heavier layers (depth, optical flow, semantic segmentation, body pose, 3D) are strictly worse on that budget.

So every heavy layer got a queue instead of a longer timeout. Ten of the twelve catalogued layers run as jobs; captions and object labels are cheap enough to stay synchronous. Which is which is recorded in the layer catalog.

The existing synchronous per-episode endpoints were not removed. The queue is a lane beside them.

One further job layer is not an annotation layer at all: annotation_review grades the annotations that already exist rather than producing any. See the automated review pass.

Enqueue a job

curl -fsS -X POST -H "Authorization: Bearer $EULER_TOKEN" \
  -H "Content-Type: application/json" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/annotation/jobs" \
  -d '{"layer_id": "depth"}'

Omitting episode_ids scopes the job to the whole project. Pass an explicit list to scope it:

curl -fsS -X POST -H "Authorization: Bearer $EULER_TOKEN" \
  -H "Content-Type: application/json" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/annotation/jobs" \
  -d '{"layer_id": "hand_pose", "episode_ids": ["ep_0007", "ep_0008"]}'

The response is the job record: id, status, progress, estimated_cost_usd, and the scope. Writes need the engineer role; reads need viewer.

The rules the enqueue enforces

Three things stop a wasted or duplicated spend, in this order.

Two further rules reject a request outright:

Re-running a layer

Euler will not silently redo work you already paid for, and it will not silently skip work you asked for. So a re-queue over recordings that already carry the layer’s output is answered explicitly.

Every recording in scope already covered. The request returns 409 with a structured detail, so a client can offer a real choice rather than an error:

{"detail": {
  "code": "already_produced",
  "layer_id": "captions",
  "message": "Scene captions already produced output for all 2 of 2 recordings in scope. Re-run with force to redo it.",
  "covered_episodes": ["ep_0007", "ep_0008"],
  "uncovered_episodes": []
}}

Only some covered. The job is narrowed to the recordings that genuinely need the work, and the ones left alone are named on the job’s covered_episodes.

You want it redone anyway. Pass force:

curl -fsS -X POST -H "Authorization: Bearer $EULER_TOKEN" \
  -H "Content-Type: application/json" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/annotation/jobs" \
  -d '{"layer_id": "captions", "force": true}'

The whole scope is redone and the job record carries forced: true, so the spend is attributable afterwards. Coverage is decided by the same checks the coverage report uses, so “already produced” here and “ran” on your dashboard can never disagree.

Watch progress

Poll

curl -fsS -H "Authorization: Bearer $EULER_TOKEN" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/annotation/jobs/$JOB_ID"

List every job for a project, newest first, and narrow with ?status= or ?layer_id=:

curl -fsS -H "Authorization: Bearer $EULER_TOKEN" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/annotation/jobs?status=running"

Stream

curl -N -H "Authorization: Bearer $EULER_TOKEN" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/annotation/jobs/stream"

The stream emits one job event per record whose state changed since the last tick, a : heartbeat comment about every 15 seconds so intermediaries keep the connection open, and a close event after about 240 seconds. That close is deliberate: it sits inside the 300-second edge budget, and a browser EventSource reconnects on its own, so a proxy timeout never looks like an error.

const stream = new EventSource(
  `${baseUrl}/v1/projects/${project}/annotation/jobs/stream`,
);
 
stream.addEventListener("job", (event) => {
  const job = JSON.parse(event.data);
  console.log(job.layer_id, job.status, job.progress.done, "/", job.progress.total);
});

Job states

StatusMeaning
queuedAccepted and waiting for a worker.
runningA worker is executing the layer’s handler.
completeFinished. result carries the handler’s summary and measured_cost_usd the metered spend.
failedThe handler raised. error carries the text.
cancelledCancelled before or during execution.

progress carries done, total and current_item, so a client can say what is being worked on rather than only how many items are left. total is 0 until the handler knows it.

The record also carries covered_episodes (recordings that already had this layer’s output when the job was queued), forced (you asked for those to be redone) and deduplicated (this response was folded onto a job that was already in flight).

Cancel

curl -fsS -X POST -H "Authorization: Bearer $EULER_TOKEN" \
  "$EULER_BASE_URL/v1/projects/$PROJECT/annotation/jobs/$JOB_ID/cancel"

Cancel is reachable from queued and running alike. A queued job flips to cancelled at once. A running job aborts at the handler’s next cancellation check, so cancellation is prompt rather than instantaneous. A job that already finished returns 409, and an unknown job returns 404.

Cancel stops the lane. It does not undo it. Recordings the job already finished keep their output, and the cancelled job’s result carries the same summary shape a completed job would, with processed_episodes naming exactly what landed. You are billed for the work that was done, and you can see what it was.

Durability

Jobs are in-memory and process-local today. A restart or a deploy loses in-flight jobs. Nothing is left half-written, because the queue owns no persistent state and the annotation output a handler produces goes through the object store as usual, but a lost job has to be re-enqueued.

Durable job records are planned to land with the Postgres store migration (tracked as T-411). Until then, treat a long job the way you would treat a long run: check it after a deploy.

Worker concurrency defaults to two per process. That is enough to keep a cold GPU start from blocking a cheap CPU lane, and small enough that one container cannot stampede the GPU lane.

Next