Skip to main content
POST
Requires version 1.1.12 or later of the Archetype platform.

Overview

This endpoint creates and starts an evaluation: the blueprint to evaluate, the examples to score it on, and — only where the run differs from what the blueprint declares — how to score them.
Only blueprints that declare a metrics schema support being evaluated. The blueprint must also be complete: a blueprint that declares values with no default (written ${required}), which only an optimization can provide, cannot be evaluated directly.
Per-target scoring overrides and the headline primary metric are resolved against the blueprint at creation, so the returned eval records exactly what will be scored. The eval is created pending and returns immediately; it moves to running and then to completed, failed, or cancelled on its own. Poll Get Eval to follow it and to read its metrics_report and output_artifacts. Use List Eval Examples to find out where it scored badly. The request is rejected with 400 when it contradicts the blueprint’s metrics schema, when a constraint bounds an objective the run does not compute, or when the blueprint is incomplete (it declares values with no default, written ${required}). It is rejected with 403 (capability_locked) when the blueprint is not available to the caller’s organization, and with 404 when the blueprint is not found. The request is rejected with 422 when a field is malformed, including a primary or a constraint objective that is not written <target>.<objective>.
Starting with version 2.0.0, a malformed primary is rejected with 422 instead of 400.
An eval of a blueprint in a project is in that project. Creating it requires being able to modify the project: you get 403 when you can see the project but not modify it, and 404 when the blueprint is in a project you cannot see. The eval’s job and output files belong to the project’s creator, whoever starts it, and every input file must be one the project’s creator can see. Any other input file returns 400. A blueprint outside any project, including a canonical blueprint, can’t be evaluated directly: the request returns 400 with code project_required. Copy the blueprint into a project with Copy Blueprint and evaluate the copy. v2.0.1+ Only each example’s inputs is required. An example following the conventions its blueprint declares needs nothing else: it is named after its input file, and each target’s labels are read from a column named after the target. State a deviation once: on the run’s targets when every example shares it, on the example itself when only one does.

Request

string
required
The blueprint to evaluate — a blp_ id. Must exist and be visible to the caller’s organization.
array
required
The examples to score. Must be non-empty. See Example object.
string
Optional human label. Defaults to an empty string.
string
The run’s headline, as <target>.<objective> — e.g. state.macro_f1. Defaults to the blueprint’s own. Must name a target this run scores and an objective it computes for it.
boolean
default:"true"
Emit the predictions NDJSON artifacts alongside the metrics report: one per target, with a row per scored item. The artifacts are what make a score explainable. Set false to skip them when the input is large enough that they are not worth their size.
array
Bounds the run reports a verdict on, replacing those the blueprint declares. Omit it to use the blueprint’s; send [] for none. Each entry names an objective the run computes, with a min, a max, or both. See Constraint. Missing a constraint does not fail the eval: Get Eval reports each constraint’s verdict and whether they all hold. v2.0.0+
object
Per-target overrides for this run, keyed by target name. Every field of an entry is optional and falls back to what the blueprint’s target declares, so a run that wants nothing different omits this entirely — and one that does need a change states it once here rather than on every file. See Target override.
targets overrides apply to every example of the run; an example may still override them for itself.
A key must name a target the blueprint declares. An entry’s objectives replaces the target’s own list for this run, and may include objectives the target does not declare.

Example object (examples[])

One thing an eval scores the agent on: the inputs handed to the agent, and the ground truth expected of it. An eval run covers several, and reports each separately as well as together — which is what lets a run say not just how good the agent is but where it is bad.
array
required
The data handed to the agent for this example, in order: file references, each file used at most once in a request. Each entry is a data ref.
Starting with version 2.0.0, an example may have several file references. Earlier versions permitted exactly one.
inputs is the only property required on an example. A conventionally-shaped example writes nothing else; it is named after its input file, and each target’s labels come from a column named after the target.
string
The name by which this example is identified in the run’s results. Defaults to the file name of its input, without the extension. Names must be unique within a request: results are keyed by name, and two examples sharing one cannot be told apart in a report.
object
The annotation supplied with this example, keyed by a name of your choosing: where its values are and, where they are not recorded at the cadence the agent emits at, how they reduce. Each value is a ground-truth declaration.This says nothing about which target it serves; that’s done by binding, so the same annotation can be written once and scored against different agents. Omit ground_truth entirely to use the convention: each target’s labels are a column of the input named after the target.
object
Which targets each annotation serves, keyed by the annotation’s name; each value is an array of target names. Omit it whenever the names line up: an annotation named after a target serves that target, and a single annotation serves a single target whatever either is called.
object
Free-form tags for this example. Carried through to its results, where they are what makes a long per-example table answer a question — which site, which shift, which operator.

Data ref (examples[].inputs[])

string
required
Storage kind, e.g. file or dataset.
string
required
Identifier within that storage kind, e.g. file_abc123.
string
Optional format hint, e.g. csv.
string
Whole-file CRC32C checksum of the referenced bytes, base64-encoded exactly as S3 emits it. You don’t need to send it: when the eval is created, it’s pinned from the checksum the data service stores for the file, so the eval records exactly which bytes it scored even if a filename is later reused. null when the data service has no checksum for the file. v1.1.13+
object
Free-form detail about this ref. Any key is carried through untouched. v1.1.13+

Ground-truth declaration (examples[].ground_truth.<name>)

One annotation supplied with an example. Both fields of this object are optional and layer over what the run already settled, because the two are independent statements: an annotation in a column of its own overrides from and leaves the reduction alone; an annotation recorded at a coarser cadence than the rest of the set overrides downsampling and leaves its location alone.
object
Where this example keeps the annotation’s values. Omitted to keep whatever the run settled on — by default, a column named after the target it serves. One of three forms:
  • {"column": "state"} — the labels are a column of the input itself, one label per record. Inputs of the same request may each name a different column. An optional values map ({"0": "idle", "1": "running"}) maps each value as it appears in the column onto the label it means, for an input that spells its labels differently — integer codes, a vendor’s own wording. Omitted means the column already holds the labels. A value the map does not cover is a read error, not a silently dropped record.
  • {"constant": "idle"} — one label covers the whole input, and every record of the input is scored against it.
  • {"per_input": {"fil_idle": "idle", "fil_drilling": "drilling"}} — each input carries a label of its own, which covers the whole of it. Keys are the ids of the example’s inputs, and values are labels of the target. An input left out has no label for this target and is not scored against it. Use this to group files already separated one per class into a single example. v2.0.0+
Unknown keys in either form are rejected rather than ignored.
string
For a target paired by_time only: how the values of the records making up one analysis window reduce to the single value that window is scored against: last_record or majority_vote. Omitted to keep the rule the run settled on.
downsampling is refused for a target paired any other way, which has nothing to reduce.

Target override (targets.<target name>)

What one run wants differently about one target. Every field is optional and falls back to the target’s own declaration. The fields here are the scoring protocol — how the agent is measured — never what the agent decides or where its decision lands, which only the blueprint may say.
array
The objectives to compute for the target in this run, replacing those the target declares. The list may include objectives the target does not declare. Each entry is one of:
  • the name of an objective the target declares, which keeps that declaration
  • a catalog objective that takes no argument, named bare: macro_f1, accuracy, balanced_accuracy, foreground_macro_f1, detection_recall, or detection_precision
  • an objective written in full, as an object with the catalog objective as its key and its arguments as the value, plus an optional name
The per-class objectives precision, recall, and f1 take a required class argument, one of the target’s labels, so they are always written in full: for example, {"name": "recall_cold", "recall": {"class": "cold"}}. name is what primary and the metrics report refer to the objective by, and defaults to the catalog objective’s name. Names must be unique within a target, so give each one a name to compute the same per-class objective for several classes. Written in full, an objective may reuse a name the target declares, and then replaces that declaration for the run. macro_f1 is shorthand for {"macro_f1": {}}.Written in full, an objective may also set if_undefined: the number it takes wherever it has no value of its own, such as an example with nothing to compute it from. For example, {"mean_iou": {}, "if_undefined": 1}. The value must be within the range of values the objective takes, or the eval is rejected when it is created. The metrics report lists the objectives that took this value in defaulted. v2.0.1+
Starting with version 2.0.0, objectives replaces the target’s list and may add objectives the target does not declare. Earlier versions could only narrow the list.
The catalog is closed: an objective outside it is rejected when the eval is created, rather than producing an unscored run.
object
Where every input of this run keeps the target’s labels, when they all agree and disagree with the target’s name. An input may still override it. Same forms as a ground-truth declaration’s from.
string
For a target paired by_time only: how the values of one analysis window reduce to the single value it is scored against, for every example of this run: last_record or majority_vote. An example may still override it.
downsampling is refused for a target paired any other way.

Constraint (constraints[])

A bound on one objective the run computes: a floor, a ceiling, or both. v2.0.0+
string
required
The objective bounded, as <target>.<objective>, naming an objective the run computes for that target, with targets applied. For example, state.macro_f1, or state.recall_idle for a per-class objective added through targets.
number
The lowest acceptable value, inclusive.
number
The highest acceptable value, inclusive.

Response

Returns 201 Created with the eval in pending status, in the same format Get Eval returns.
string
required
The eval’s evl_ id.
string
required
Human label given on create. Empty when none was given.
string
required
The organization that owns the eval.
string
required
The blueprint being evaluated.
string
required
The run’s headline, as <target>.<objective> — resolved at creation, so this is what was actually scored rather than what was asked for.
object
required
The objectives this run computes, keyed by target name — the blueprint’s own declaration folded together with the request’s per-target overrides, resolved at creation. Each objective is in the same form as in a target override: a bare name, or written in full with its arguments. primary names one of these, as <target>.<name>. Read this rather than the blueprint’s metrics to know what was scored, since a request may replace a target’s objectives.
array
required
The bounds the run reports a verdict on, resolved at creation: the request’s, or else the blueprint’s. Empty when there are none. v2.0.0+
boolean
required
Whether the run emits the predictions artifacts alongside the metrics report, one per target, as the request asked.
array
required
The examples the eval was created against, as resolved: each carries the name its results are keyed by, and inputs stamped with the CRC32C of the bytes scored. See Resolved example object.
string
required
The eval’s current lifecycle status; pending on creation.
string
The project the eval is in: its blueprint’s. This value is null outside any project. v2.0.0+
string
required
Subject id (usr_... or key_...) that created this eval.
string
required
When the eval was created (date-time).
string
When scoring began. Absent while pending.
string
When the eval reached a terminal status. Absent until then.
object
What the run measured. Present once the eval is completed. See Get Eval for its shape.
array
Refs to the files the run produced, filled in while the eval runs. Each id is a data-service file id, so an artifact is downloadable through the files API.
string
Failure detail; null unless the eval failed.

Resolved example object (examples[])

string
required
How this example is identified in the run’s results.
Example names must be unique within a request, because results are keyed by name.
integer
required
The example’s 1-based position in the run, fixing the order results are listed in.
array
required
The data handed to the agent, each ref carrying the CRC32C of the bytes scored. See Resolved input ref for the keys the service writes into each ref’s metadata.
object
The free-form tags the example was created with, if any.

Resolved input ref (examples[].inputs[])

Each input of a returned example is a data ref, with the same type, id, and format as in the request, plus the crc32c of the bytes scored. Its metadata carries the following keys, written by the service when the eval is created, never by the caller. Any other key is carried through untouched.
object
Where this input’s ground truth is for every target the eval scores, with nothing left to convention: a map of target name to that target’s resolved ground truth.
string
The name of the example the input belongs to. v1.1.13+

Resolved ground truth (examples[].inputs[].metadata.targets.<target name>)

A ground truth with nothing left to convention: where its values are, and how they reduce. This is what an example’s ground-truth declaration becomes once laid over the defaults the run settled on.
object
required
Where the values are. Same forms as a ground-truth declaration’s from.
string
For a target paired by_time, how the values of the records making up one analysis window reduce to the single value that window is scored against: the target’s own rule, unless the run or the example overrides it. Absent for a target paired any other way, which has nothing to reduce.