Skip to main content
GET
Requires version 1.1.12 or later of the Archetype platform.

Overview

This endpoint returns one eval by its evl_ id. An eval in a project you cannot see returns 404. Fields beyond the always-present ones are populated as the eval progresses:
  • started_at is populated when scoring begins.
  • output_artifacts fills in while the eval runs. An entry whose metadata.status is partial holds the rows scored so far, which is also what a failed or cancelled mid-run eval leaves behind.
  • metrics_report is populated once the eval is completed.
  • completed_at is populated when the eval reaches a terminal status.
  • error is populated when the eval failed.
The metrics_report answers the question “is this agent good enough?” To answer “where is this agent bad?” page through List Eval Examples.
The HTTP status code returned is 404 both if the specified eval ID is unknown and if the ID exists but belongs to another organization.

Request

string
required
Eval evl_ id.

Response

string
required
The eval’s evl_ id.
string
required
Human label given on create. Empty when none was given.
string
required
The organization that owns the eval.
string
required
The blueprint being evaluated.
string
required
The run’s headline, as <target>.<objective> — resolved at creation, so this is what was actually scored rather than what was asked for.
object
required
The objectives this run computes, keyed by target name; each value is an array of objectives, each a bare name or written in full with its arguments, as in a target override. This is the blueprint’s own declaration folded together with the request’s per-target overrides, resolved at creation. primary names one of these, as <target>.<name>.
examples records what the run actually read; that is, resolved names, the CRC32C of the bytes scored, and fully-resolved ground-truth declarations, rather than what the request asked for.
Each input’s metadata carries the service-written targets and example_name keys; see Resolved input ref.
array
required
The bounds the run reports a verdict on, resolved at creation: the request’s, or else the blueprint’s. Empty when there are none. See Constraint. v2.0.0+
boolean
required
Whether the run emits the predictions artifacts alongside the metrics report, one per target, as the request asked. When true, they appear in output_artifacts as the run produces them.
array
required
The examples the eval was created against, as resolved: each carries the name its results are keyed by, and inputs stamped with the CRC32C of the bytes scored. See Resolved example object for the format of the objects in this array.
string
required
The eval’s current lifecycle status: pending, running, completed, failed, or cancelled.
Evals are always created with the status pending. Once they’re dispatched, their state changes to running. From there, it will eventually enter one of the three terminal states: completed, failed, or cancelled. A pending eval can also be cancelled directly.
string
The project the eval is in: its blueprint’s. This value is null outside any project. v2.0.0+
string
required
Subject id (usr_... or key_...) that created this eval.
string
required
When the eval was created (date-time).
string
When scoring began (date-time). Absent while pending.
string
When the eval reached a terminal status (date-time). Absent until then.
object
What the run measured: its headline score and what was measured for each target. Present once the eval is completed; absent while it is pending or running, and on a cancelled or failed eval. See Metrics report for the format of this object.
array
Refs to the files the run produced. Each id is a data-service file id, so an artifact is downloadable through the files API, and each carries metadata saying which artifact it is. See Output artifact ref for the format of the objects in this array.Populated while the run goes, not only at the end: each batch of scored rows is appended to the predictions file and updates the entry, whose metadata.status stays partial until the run’s final push marks it complete. An eval that failed or was cancelled mid-run keeps its partial entry — the rows scored so far are still readable.
string
Why the eval failed. Present only when status is failed.

Resolved example object (examples[])

One example of a run, as the run recorded it: named, positioned, and with its inputs pinned to the bytes it read.
string
required
How this example is identified in the run’s results. Unique within the eval.
integer
required
The example’s 1-based position in the run, fixing the order results are listed in.
array
required
The data handed to the agent, each ref carrying the CRC32C of the bytes scored, in its crc32c field.
object
The ground truth the example was created with, if it declared any. Same shape as the ground_truth of a request example.
object
Which targets each of the example’s ground truths serves, as it was created with, if it said.
object
The free-form tags the example was created with, if any.

Metrics report (metrics_report)

What an eval run measured over the whole run: its headline score, and what was measured for each target. schema_version says which fields are present. Reports from v2 on carry the example counts, the headline’s aggregation, and each target’s pairing, item counts, and per-example summary; v1 reports carry none of them.
string
required
The report’s schema version, v1 or v2.
object
required
The run’s headline score. Repeats a value that also appears in the named target’s aggregate, so a reader wanting the one number the run is judged by gets it without following a pointer.
  • target (string, required) — the target this headline is about.
  • name (string, required) — the objective’s name, e.g. macro_f1.
  • value (number) — its value, or null when nothing was scored.
  • aggregation (string) — how the value was computed from the run’s examples: pooled or per_example. From v2.
object
required
One entry per scored target, keyed by target name. See Target report for the format of a target record.
array
What the run measured against each of its constraints, over the whole run. See Constraint verdict. Absent when the run has no constraints. Missing a constraint does not fail the run. v2.0.0+
boolean
Whether every constraint holds: false when one is missed, null when none is missed but one cannot be judged because its objective has no value. Present whenever constraints is. v2.0.0+
integer
How many examples the run has. From v2.
integer
How many examples could not be scored at all, and are left out of every figure. From v2.
integer
How many examples were scored, but with part of their ground truth unusable. From v2.
integer
How many ground-truth records, over every example scored, could not be matched to any prediction and were dropped. An example that failed is not counted. From v2.

Constraint verdict (metrics_report.constraints[])

What a run measured against one of its constraints. v2.0.0+
object
required
The constraint judged: its objective, and its min, max, or both.
number
The objective’s value. null when it has none, as when nothing it reads was scored.
boolean
Whether the value respects the bounds. null when there is no value to judge.

Target report (metrics_report.targets.<target name>)

What was measured for one target, over the whole run or over one example. The fields after aggregate depend on the target’s type.
object
required
Every objective computed for the target, by name, over what the report covers: for the whole run, computed as the headline’s aggregation says. A value is null when nothing was scored.
object
What each objective was declared with, by name, for every objective declared with arguments or under a name other than its catalog objective’s: its arguments, plus the catalog objective as objective when the name is not its own. For example, {"recall_anomaly": {"objective": "recall", "class": "anomaly"}}. v2.0.0+
Starting with version 2.0.1, an objective declared with an if_undefined is listed here too, with its if_undefined.
array
The objectives in aggregate that have no value of their own and report the value they were declared to take in that case (their if_undefined) instead. Over the whole run, only objectives computed pooled are listed; for an objective computed per_example, see per_example.defaulted. Absent when empty. v2.0.1+
string
required
The target’s type: category (one class per scored item) or binary (one of two classes per scored item, one of them its positive class). binary is from v2; reports of schema v1 also describe a binary target as category.
array
required
The class vocabulary, fixing the order of every per-class list and of the confusion matrix.
object
required
Per-class figures, each a list in class_names order. A figure is null where there is nothing to compute it from, rather than 0.
Starting with version 2.0.0, a per-class figure with nothing to compute it from is null. Earlier versions reported 0.
  • precision (array of numbers) — of the items predicted as the class, the fraction that truly were. null for a class never predicted.
  • recall (array of numbers) — of the items truly of the class, the fraction predicted as it. null for a class that never occurs.
  • f1 (array of numbers) — harmonic mean of precision and recall, per class. null for a class neither occurring nor predicted.
  • support (array of integers) — how many items truly of the class were scored.
array
required
Row = true class, column = predicted class, both in class_names order. Counts the items where both a prediction and a ground truth were found.
string
binary targets only, and required for them: the class standing for the condition the target detects, one of class_names.
string
category targets only: the class meaning that nothing happened, one of class_names, when the target names one. This is the class the foreground and detection objectives set apart.
array
Per true class, the ground truths that got no prediction. From v2.
array
Per predicted class, the predictions that had no ground truth. From v2.
object
How many scored items ended in each outcome, for a target whose pairing can leave a prediction or a ground truth unmatched. From v2.
  • matched (integer, required) — items with both a prediction and a truth.
  • missed (integer, required) — truths with no prediction.
  • false_alarms (integer, required) — predictions with no truth.
  • unscored (integer, required) — predictions outside the annotated coverage, counted in no denominator.
  • failed (integer, required) — items whose ground truth could not be read, counted in no denominator.
string
How the target’s predictions were paired with its ground truth into scored items. Currently always by_time. From v2.
integer
How many items were scored. From v2.
integer
How many of those items could not be judged, their ground truth being unreadable. From v2.
object
Over the whole run only: each objective’s per-example values summarized, keyed by objective name. Only the examples where the objective has a value count, including the value it takes from its if_undefined. An objective’s summary is null when no example has a value for it. From v2.
  • mean (number, required) — the mean of the values.
  • min (number, required) — the lowest value.
  • median (number, required) — the median value; the mean of the two middle ones for an even count.
  • max (number, required) — the highest value.
  • p90 (number, required) — the 90th percentile, interpolating linearly between values.
  • excluded (integer, required) — how many examples had no value, and are left out of every figure here.
  • defaulted (integer) — how many examples had no value of their own and count with the objective’s if_undefined value in every figure here. Absent when none did. v2.0.1+
  • stderr (number) — the standard error of the mean, or null with fewer than two values.

Output artifact ref (output_artifacts[])

string
required
Storage kind, e.g. file.
string
required
Data-service file id, so the artifact is downloadable through the files API.
string
Optional format hint.
string
Whole-file CRC32C checksum of the referenced bytes, base64-encoded exactly as S3 emits it. null when the data service has no checksum for the file.
object
For one of an eval’s predictions files: which file it is, and how complete.
  • kind (string, required) — which artifact this is among the eval’s outputs: predictions for a predictions file.
  • status (string, required) — complete when the file holds every scored item, or partial when it holds only the items scored so far.
  • row_count (integer, required) — rows in the file: how many scored items a reader will find in it, one per row.
  • target (string) — the target whose predictions the file holds: an eval writes one predictions file per target it scores. Absent on a file from an eval that wrote a single predictions file for all its targets.

Important Notes

The run-level figure is every example pooled, never the mean of the per-example results from List Eval Examples.