Data utility
A single readiness number answers three questions at once, and they are not the same question. Was the recording captured correctly? Do the labels describe what actually happened? Will this data help the model you are about to train?
Those have different owners and different fixes, so Euler reports them as three separate layers, and reports what it has not measured just as plainly as what it has.
Anything unmeasured is reported as not assessable, with the reason and what would unlock it. Euler never reports a missing measurement as a zero. A missing embedding index is not poor coverage, and an unlinked failure is not irrelevant data. The two look identical in a number and completely different in a decision.
The three layers
| Layer | The question it answers | What you do when it is weak |
|---|---|---|
| Integrity | Was this recording captured correctly? | Fix the rig, the clocks or the capture script, then recollect |
| Semantic quality | Do the labels describe what actually happened? | Re-annotate, or raise the bar your annotation layers have to clear |
| Utility | Will this data help the model you are about to train? | Collect a different slice, or curate a different subset |
Read them with:
curl -fsS -H "Authorization: Bearer $EULER_TOKEN" \
"$EULER_BASE_URL/v1/projects/$PROJECT/utility"The response carries each layer, every signal inside it, and the headline readiness score unchanged so you can reconcile the layers against it. Nothing is recomputed here: the integrity and semantic numbers are the same measurements behind your readiness score, read through your own project’s settings. A layer can never disagree with the headline.
Every signal also says which layer it sits in and why, so you can argue with the placement rather than guess at it.
Utility has no single score
The utility layer reports its signals separately and gives you no blended number. That is deliberate.
Blending them would need weights, and nobody has calibrated those weights against a measured training outcome. A utility score that has never been validated against a real training result would be a guess with a decimal point on it, and it is exactly the claim we were asked not to make. When a measured probe run exists to calibrate it, a blend becomes possible. Until then the absence is the honest answer.
The utility signals
| Signal | What it means | Available once |
|---|---|---|
| Policy fit | The share of your reviewed recordings that your declared training target can actually consume. A file a trainer rejects is not training data. | You declare a target profile |
| Distribution spread | How much of the content space this data covers. A tight cluster trains a policy that only works in that cluster. | The project is indexed |
| Near duplicate rate | Near-identical recordings as a share of all pairs. A high rate means the dataset is smaller than its episode count suggests. | The project is indexed |
| Failure relevance | How much of this data sits near failures your model actually made. This is the data most likely to change an evaluation. | You link model feedback |
| Predicted downstream value | Whether these scores actually predict that a model trained on this data gets better. | You supply a measured probe run |
Failure relevance counts only failures your model reported. Euler’s own candidate flags are not model feedback, and they are never reported as if your model had failed there.
Predicted downstream value has no path to a value other than a measured training run you supply. Euler does not predict a training improvement it has not measured.
The proxy metrics under semantic quality
These are how Euler checks its own annotation work. Each one carries its plain definition, how it was measured, what was excluded, and how many units the number is over.
Temporal segment overlap
How closely Euler’s action segments line up in time with a human-adjudicated segmentation of the same recording.
Each predicted segment is matched to at most one adjudicated segment, best overlap first, and every unmatched prediction counts against the score. Splitting one real step into many therefore lowers the number rather than raising it, which is the point: a method that emits a boundary every half second scores near zero.
Not comparable to published action-segmentation scores. Euler matches on time alone. Published scores also require the labels to match, and Euler’s step labels are free text rather than draws from a fixed taxonomy, so requiring a label match would measure spelling rather than timing. Label agreement is reported as its own separate number. Do not put a Euler segment score next to a published benchmark figure.
Boundary agreement
How often Euler puts a step boundary within a short window of where a human put one, balanced against how many boundaries it invents. You choose the window:
curl -fsS -H "Authorization: Bearer $EULER_TOKEN" \
"$EULER_BASE_URL/v1/projects/$PROJECT/utility?boundary_tolerance_s=0.25"The clip’s own first and last edges are excluded, because every method gets those right for free and counting them would inflate the number.
Language and action agreement
The share of caption windows where an independent referee agreed that the written language describes what the frames show. Windows the referee could not confidently judge are excluded from the ratio and counted separately, rather than being folded in as agreement.
Annotation confidence
The confidence the segmenter reported about its own output. This is what the model claimed, not evidence that it was right, and it is always labelled that way.
Why these need a gold set
Segment overlap and boundary agreement are only reported when a human-adjudicated
segmentation is stored for the recording. Without one they read not assessable
with that reason.
Euler will not score its own segmentation against itself. Your reviewers’ edits replace the machine output rather than sitting beside it, so scoring the stored output would compare a thing to itself and report a perfect number on every reviewed recording. A separate adjudicated record is the only honest reference, and no record means no measurement.
Raw against filtered against curated
The real question about a data product is not “what is my score”, it is “does acting on this score change anything”. Downstream training feedback answers that and takes months, so Euler runs the controlled comparison instead: hold the model and the compute fixed, vary only the data, and report what each subset contains.
curl -fsS -X POST -H "Authorization: Bearer $EULER_TOKEN" \
-H "Content-Type: application/json" \
"$EULER_BASE_URL/v1/projects/$PROJECT/ablation" \
-d '{}'| Arm | What it is | What it isolates |
|---|---|---|
| Raw | Every recording in the project, nothing removed | The baseline any claim has to beat |
| Readiness filtered | One stage only: the readiness floor | What the score alone buys you |
| Euler curated | The full recommended plan: floor, then duplicates, then balance, then coverage | The whole product |
The readiness floor is held constant across the filtered and curated arms, so the difference between them is exactly what curation does beyond the score. Each arm reports its size, its readiness distribution, its duplicate rate, how many failure-linked recordings survive, which faults it still carries and its composition across your facets. The curated arm carries the same per-stage receipt the curation screen shows, because it is the same engine.
Nothing is saved. The comparison needs read access only, exactly like a curation preview.
This is composition, not a training result. The comparison tells you what each subset contains. It does not claim your model trains better on one of them, and it will not, unless you supply a measured probe-training run with the request. Composition is evidence. Conflating it with a training outcome is the claim this whole page exists to avoid.
Supplying a measured run flips the report’s own claim from not assessable to
measured, and your numbers are reported as yours:
curl -fsS -X POST -H "Authorization: Bearer $EULER_TOKEN" \
-H "Content-Type: application/json" \
"$EULER_BASE_URL/v1/projects/$PROJECT/ablation" \
-d '{
"probe": {
"measured": true,
"model_description": "3-seed BC probe, identical schedule per arm",
"metric_name": "success_rate",
"raw_score": 0.41,
"filtered_score": 0.47,
"curated_score": 0.52
}
}'What this dataset does not cover
curl -fsS -H "Authorization: Bearer $EULER_TOKEN" \
"$EULER_BASE_URL/v1/projects/$PROJECT/distribution-gap"Gaps are ranked in the order a team should act on them, and each one names what kind of work fixes it.
| Gap | What it means | What to do |
|---|---|---|
| Fault concentrated | Most recordings of this slice carry a readiness fault | Fix the capture problem first. Collecting more under the same setup repeats the fault |
| Lost in curation | The dataset has this slice and the curated subset keeps none of it | Check why every recording was dropped, then loosen the stage or recollect it |
| Failure uncovered | Your model failed here and almost nothing in the project sits near that failure | Collect more of this situation. It is the data most likely to change your next evaluation |
| Thin coverage | Too few recordings to train on or evaluate on | Collect more of this slice |
Every gap names something that exists in your recorded data. Euler cannot know about a situation nobody has ever recorded, and it says so rather than inventing a category. When nothing crosses the bar, the report says nothing stood out and adds that this is not proof of full coverage.
Signals the report wanted and did not have are listed too. Without an index, gaps are reported over your recorded facets rather than over the content space. Without linked model feedback, Euler cannot tell you which gaps matter to a failure you actually saw.
Next
- What the individual checks measure: Readiness.
- Selecting the subset the curated arm describes: Curation.
- What the annotation layers produced, and how it was graded: Automated review pass.