{ "errors": [ { "code": "capability_locked", "message": "Blueprint not available to the caller's organization.", "suggestion": null, "error_uid": "err-xxxxxxxx" } ]}
{ "errors": [ { "code": "capability_locked", "message": "Blueprint not available to the caller's organization.", "suggestion": null, "error_uid": "err-xxxxxxxx" } ]}
Requires version 1.1.12 or later of the Archetype platform.
Overview
This endpoint creates and starts an evaluation: the blueprint to evaluate, the examples to score
it on, and — only where the run differs from what the blueprint declares — how to score them.
Only blueprints that declare a metrics schema support being evaluated. The blueprint must also
be complete: a blueprint that declares values with no default (written ${required}), which
only an optimization can provide, cannot be evaluated directly.
Per-target scoring overrides and the headline primary metric are resolved against the
blueprint at creation, so the returned eval records exactly what will be scored.The eval is created pending and returns immediately; it moves to running and then to
completed, failed, or cancelled on its own. Poll Get
Eval to follow it and to read its metrics_report and
output_artifacts. Use List Eval Examples to
find out where it scored badly.The request is rejected with 400 when it contradicts the blueprint’s metrics schema, when a
constraint bounds an objective the run does not compute, or when the blueprint is incomplete (it
declares values with no default, written ${required}). It is rejected with 403
(capability_locked) when the blueprint is not available to the caller’s organization, and with
404 when the blueprint is not found.The request is rejected with 422 when a field is malformed, including a primary or a
constraint objective that is not written <target>.<objective>.
Starting with version 2.0.0, a malformed primary is rejected
with 422 instead of 400.
An eval of a blueprint in a project is in that project. Creating
it requires being able to modify the project: you get 403 when you can see the project but not
modify it, and 404 when the blueprint is in a project you cannot see. The eval’s job and output
files belong to the project’s creator, whoever starts it, and every input file must be one the
project’s creator can see. Any other input file returns 400.A blueprint outside any project, including a canonical blueprint, can’t be evaluated directly: the
request returns 400 with code project_required. Copy the blueprint into a project with Copy
Blueprint and evaluate the copy. v2.0.1+Only each example’s inputs is required. An example following the conventions its blueprint
declares needs nothing else: it is named after its input file, and each target’s labels are read
from a column named after the target. State a deviation once: on the run’s targets when every
example shares it, on the example itself when only one does.
Request
string
required
The blueprint to evaluate — a blp_ id. Must exist and be visible to the caller’s organization.
array
required
The examples to score. Must be non-empty. See Example object.
string
Optional human label. Defaults to an empty string.
string
The run’s headline, as <target>.<objective> — e.g. state.macro_f1. Defaults to the
blueprint’s own. Must name a target this run scores and an objective it computes for it.
boolean
default:"true"
Emit the predictions NDJSON artifacts alongside the metrics report: one per target, with a row
per scored item. The artifacts are what make a score explainable. Set false to skip them when
the input is large enough that they are not worth their size.
array
Bounds the run reports a verdict on, replacing those the blueprint declares. Omit it to use the
blueprint’s; send [] for none. Each entry names an objective the run computes, with a min, a
max, or both. See Constraint. Missing a constraint does not fail
the eval: Get Eval
reports each constraint’s verdict and whether they all hold. v2.0.0+
object
Per-target overrides for this run, keyed by target name. Every field of an entry is optional
and falls back to what the blueprint’s target declares, so a run that wants nothing different
omits this entirely — and one that does need a change states it once here rather than on every
file. See Target override.
targets overrides apply to every example of the run; an example may still override them
for itself.
A key must name a target the blueprint declares. An entry’s objectives replaces the target’s
own list for this run, and may include objectives the target does not declare.
Example object (examples[])
One thing an eval scores the agent on: the inputs handed to the agent, and the ground truth
expected of it. An eval run covers several, and reports each separately as well as together —
which is what lets a run say not just how good the agent is but where it is bad.
array
required
The data handed to the agent for this example, in order: file references, each file used at
most once in a request. Each entry is a data ref.
Starting with version 2.0.0, an example may have several file references. Earlier versions
permitted exactly one.
inputs is the only property required on an example. A conventionally-shaped example writes
nothing else; it is named after its input file, and each target’s labels come from a column
named after the target.
string
The name by which this example is identified in the run’s results. Defaults to the file name of
its input, without the extension. Names must be unique within a request: results are keyed by
name, and two examples sharing one cannot be told apart in a report.
object
The annotation supplied with this example, keyed by a name of your choosing: where its values
are and, where they are not recorded at the cadence the agent emits at, how they reduce. Each
value is a ground-truth declaration.This says nothing about which target it serves; that’s done by binding, so the same
annotation can be written once and scored against different agents. Omit ground_truth
entirely to use the convention: each target’s labels are a column of the input named after the
target.
object
Which targets each annotation serves, keyed by the annotation’s name; each value is an array of
target names. Omit it whenever the names line up: an annotation named after a target serves
that target, and a single annotation serves a single target whatever either is called.
object
Free-form tags for this example. Carried through to its results, where they are what makes a
long per-example table answer a question — which site, which shift, which operator.
Data ref (examples[].inputs[])
string
required
Storage kind, e.g. file or dataset.
string
required
Identifier within that storage kind, e.g. file_abc123.
string
Optional format hint, e.g. csv.
string
Whole-file CRC32C checksum of the referenced bytes, base64-encoded exactly as S3 emits it. You
don’t need to send it: when the eval is created, it’s pinned from the checksum the data service
stores for the file, so the eval records exactly which bytes it scored even if a filename is
later reused. null when the data service has no checksum for the file. v1.1.13+
object
Free-form detail about this ref. Any key is carried through untouched. v1.1.13+
One annotation supplied with an example. Both fields of this object are optional and layer over
what the run already settled, because the two are independent statements: an annotation in a
column of its own overrides from and leaves the reduction alone; an annotation recorded at a
coarser cadence than the rest of the set overrides downsampling and leaves its location alone.
object
Where this example keeps the annotation’s values. Omitted to keep whatever the run settled on —
by default, a column named after the target it serves. One of three forms:
{"column": "state"} — the labels are a column of the input itself, one label per record.
Inputs of the same request may each name a different column. An optional values map
({"0": "idle", "1": "running"}) maps each value as it appears in the column onto the label
it means, for an input that spells its labels differently — integer codes, a vendor’s own
wording. Omitted means the column already holds the labels. A value the map does not cover
is a read error, not a silently dropped record.
{"constant": "idle"} — one label covers the whole input, and every record of the input is
scored against it.
{"per_input": {"fil_idle": "idle", "fil_drilling": "drilling"}} — each input carries a
label of its own, which covers the whole of it. Keys are the ids of the example’s inputs,
and values are labels of the target. An input left out has no label for this target and is
not scored against it. Use this to group files already separated one per class into a single
example. v2.0.0+
Unknown keys in either form are rejected rather than ignored.
string
For a target paired by_time only: how the values of the records making up one analysis window
reduce to the single value that window is scored against: last_record or majority_vote.
Omitted to keep the rule the run settled on.
downsampling is refused for a target paired any other way, which has nothing to reduce.
Target override (targets.<target name>)
What one run wants differently about one target. Every field is optional and falls back to the
target’s own declaration. The fields here are the scoring protocol — how the agent is measured —
never what the agent decides or where its decision lands, which only the blueprint may say.
array
The objectives to compute for the target in this run, replacing those the target declares. The
list may include objectives the target does not declare. Each entry is one of:
the name of an objective the target declares, which keeps that declaration
a catalog objective that takes no argument, named bare: macro_f1, accuracy,
balanced_accuracy, foreground_macro_f1, detection_recall, or detection_precision
an objective written in full, as an object with the catalog objective as its key and its
arguments as the value, plus an optional name
The per-class objectives precision, recall, and f1 take a required class argument, one
of the target’s labels, so they are always written in full: for example,
{"name": "recall_cold", "recall": {"class": "cold"}}. name is what primary and the metrics
report refer to the objective by, and defaults to the catalog objective’s name. Names must be
unique within a target, so give each one a name to compute the same per-class objective for
several classes. Written in full, an objective may reuse a name the target declares, and then
replaces that declaration for the run. macro_f1 is shorthand for {"macro_f1": {}}.Written in full, an objective may also set if_undefined: the number it takes wherever it has
no value of its own, such as an example with nothing to compute it from. For example,
{"mean_iou": {}, "if_undefined": 1}. The value must be within the range of values the
objective takes, or the eval is rejected when it is created. The metrics report lists the
objectives that took this value in defaulted. v2.0.1+
Starting with version 2.0.0, objectives replaces the target’s list and may add objectives the
target does not declare. Earlier versions could only narrow the list.
The catalog is closed: an objective outside it is rejected when the eval is created, rather
than producing an unscored run.
object
Where every input of this run keeps the target’s labels, when they all agree and disagree with
the target’s name. An input may still override it. Same forms as a ground-truth
declaration’s from.
string
For a target paired by_time only: how the values of one analysis window reduce to the single
value it is scored against, for every example of this run: last_record or majority_vote. An
example may still override it.
downsampling is refused for a target paired any other way.
Constraint (constraints[])
A bound on one objective the run computes: a floor, a ceiling, or both. v2.0.0+
string
required
The objective bounded, as <target>.<objective>, naming an objective the run computes for that
target, with targets applied. For example, state.macro_f1, or state.recall_idle for a
per-class objective added through targets.
number
The lowest acceptable value, inclusive.
number
The highest acceptable value, inclusive.
Response
Returns 201 Created with the eval in pending status, in the same format
Get Eval returns.
string
required
The eval’s evl_ id.
string
required
Human label given on create. Empty when none was given.
string
required
The organization that owns the eval.
string
required
The blueprint being evaluated.
string
required
The run’s headline, as <target>.<objective> — resolved at creation, so this is what was
actually scored rather than what was asked for.
object
required
The objectives this run computes, keyed by target name — the blueprint’s own declaration folded
together with the request’s per-target overrides, resolved at creation. Each objective is in the
same form as in a target override: a bare name, or
written in full with its arguments. primary names one of these, as <target>.<name>. Read this
rather than the blueprint’s metrics to know what was scored, since a request may replace a
target’s objectives.
array
required
The bounds the run reports a verdict on, resolved at creation: the request’s, or else the
blueprint’s. Empty when there are none. v2.0.0+
boolean
required
Whether the run emits the predictions artifacts alongside the metrics report, one per target,
as the request asked.
array
required
The examples the eval was created against, as resolved: each carries the name its results are
keyed by, and inputs stamped with the CRC32C of the bytes scored. See Resolved example
object.
string
required
The eval’s current lifecycle status; pending on creation.
string
The project the eval is in: its blueprint’s. This value is null outside any
project. v2.0.0+
string
required
Subject id (usr_... or key_...) that created this eval.
string
required
When the eval was created (date-time).
string
When scoring began. Absent while pending.
string
When the eval reached a terminal status. Absent until then.
object
What the run measured. Present once the eval is completed. See
Get Eval for its shape.
array
Refs to the files the run produced, filled in while the eval runs. Each id is a data-service
file id, so an artifact is downloadable through the files API.
string
Failure detail; null unless the eval failed.
Resolved example object (examples[])
string
required
How this example is identified in the run’s results.
Example names must be unique within a request, because results are keyed by name.
integer
required
The example’s 1-based position in the run, fixing the order results are listed in.
array
required
The data handed to the agent, each ref carrying the CRC32C of the bytes scored. See Resolved
input ref for the keys the service writes into each ref’s
metadata.
object
The free-form tags the example was created with, if any.
Resolved input ref (examples[].inputs[])
Each input of a returned example is a data ref, with the same type, id, and format as in the
request, plus the crc32c of the bytes scored. Its metadata carries the following keys, written
by the service when the eval is created, never by the caller. Any other key is carried through
untouched.
object
Where this input’s ground truth is for every target the eval scores, with nothing left to
convention: a map of target name to that target’s resolved ground
truth.
string
The name of the example the input belongs to. v1.1.13+
Resolved ground truth (examples[].inputs[].metadata.targets.<target name>)
A ground truth with nothing left to convention: where its values are, and how they reduce. This is
what an example’s ground-truth declaration
becomes once laid over the defaults the run settled on.
object
required
Where the values are. Same forms as a ground-truth declaration’s from.
string
For a target paired by_time, how the values of the records making up one analysis window
reduce to the single value that window is scored against: the target’s own rule, unless the run
or the example overrides it. Absent for a target paired any other way, which has nothing to
reduce.
{ "errors": [ { "code": "capability_locked", "message": "Blueprint not available to the caller's organization.", "suggestion": null, "error_uid": "err-xxxxxxxx" } ]}
{"docsConfig":{"theme":"mint","$schema":"https://mintlify.com/docs.json","name":"Archetype AI Documentation","colors":{"primary":"#0047FF","light":"#9EBBFF","dark":"#0047FF"},"logo":{"light":"https://mintcdn.com/archetypeai-bd6fb3cf/Fml2i7FmPyxjLCP7/logo/light.svg?fit=max\u0026auto=format\u0026n=Fml2i7FmPyxjLCP7\u0026q=85\u0026s=4b1f94a0bc6c65c3905d1a55d725b05f","dark":"https://mintcdn.com/archetypeai-bd6fb3cf/Fml2i7FmPyxjLCP7/logo/dark.svg?fit=max\u0026auto=format\u0026n=Fml2i7FmPyxjLCP7\u0026q=85\u0026s=b652785f67f97925eb5177264ffc9e50"},"favicon":"https://mintcdn.com/archetypeai-bd6fb3cf/PEoHdVWOSWJmHgSg/favicon.svg?fit=max\u0026auto=format\u0026n=PEoHdVWOSWJmHgSg\u0026q=85\u0026s=4ea6587ffdd8e925003e9e12319702aa","api":{"mdx":{"auth":{"method":"bearer"},"server":"https://platform.u1.archetypeai.app/api"}},"appearance":{"default":"system"},"navbar":{"links":[{"href":"https://platform.u1.archetypeai.app/","label":"Console"}]},"footer":{"socials":{"twitter":"https://x.com/physicalai","github":"https://github.com/archetypeai","linkedin":"https://www.linkedin.com/company/archetype-ai"}},"search":{"prompt":"Search Archetype Platform docs..."},"seo":{"indexing":"navigable"},"styling":{"codeblocks":"system","latex":false},"integrations":{"amplitude":{"apiKey":"84e8a1610cf8b3c7d8a49d945917606d"}},"markdown":{"instructions":"Start with /introduction/getting-started. Use the Inference API's Direct Query endpoint (POST /api/inference/query) with the newton-c-2.6 or newton-fusion-1.0 model (text, image, and video reasoning) or the omega-embedding encoder (time-series embeddings). ATAI_API_ENDPOINT is the base URL https://platform.u1.archetypeai.app with no version path; every endpoint path begins with /api. Pages whose descriptions are marked (Archived) document the legacy Lens runtime — do not use them for new projects."},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"archetypeai-bd6fb3cf","hasAudioNarration":false,"actualSubdomain":"archetypeai-bd6fb3cf","customDomains":["docs.archetypeai.app"],"gitSource":{"type":"github","owner":"archetypeai","repo":"mintlify-docs","deployBranch":"main","contentDirectory":"","isPrivate":true,"mountPath":"","label":"","_id":"6ae399f43888be9481e90634"},"feedback":{"thumbs":true},"entitlements":{"AI_CHAT":{"status":"ENABLED"},"CONTEXTUAL_FEEDBACK":{"status":"ENABLED"},"CODE_SNIPPET_FEEDBACK":{"status":"ENABLED"},"WEB_MCP":{"status":"ENABLED"},"NARRATION":{"status":"ENABLED"}},"buildId":"6afb61d4ffd86b9df9e42a1b","clientVersion":"0.0.4227","searchFilterCounts":[{"language":"default","tag":"API Reference","count":272},{"language":"default","tag":"Documentation","count":82},{"language":"default","tag":"Agents API","count":55},{"language":"default","tag":"Archive","count":28},{"language":"default","tag":"Core Concepts","count":25},{"language":"default","tag":"Agents","count":24},{"language":"default","tag":"Batch Processing API","count":20},{"language":"default","tag":"Developer Console","count":18},{"language":"default","tag":"Files API","count":16},{"language":"default","tag":"Sessions","count":14}]},"assistantConfig":{"deflection":{"enabled":true,"email":"[email protected]"}},"basePath":"","hasChatPermissions":true,"hasWebMcpPermissions":true,"toggles":[{"name":"kill-isr","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-rbac-roles","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-mintlify-slack","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"agent-integrations-settings","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"rbac-expansion","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-pr-grouping","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"custom-prefix","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"analytics-v7","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"user-flows-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"no-publish-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-onboarding","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving-live","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"stripTemperatureMiddleware","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"support-platform-pylon-zendesk","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"snapshot-serving","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"instant-rollback","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications-server","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"granular-notifications","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automations-fable","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fixed-ai-action-pricing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"automation-resolve-merge-conflict","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"scoped-publishing","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"mintql","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"page-level-analytics","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-version-history-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"billing-page-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"full-page-search","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workspace-projects","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"assistant-v5","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"fork-pr-previews","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"revalidation-jwt","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-sso-scim","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"hobby-abuse-scan","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"dashboard-inbox","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-signup","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-dual-write","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"workos-enabled","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"approve-review-flow","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"editor-metadata","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"sso-setup-deadline-banner","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"turbopuffer-serve","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"live-agents","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"starter-pricing","enabled":true,"variant":{"name":"control","enabled":true,"payload":{"type":"string","value":"control"},"feature_enabled":true},"impressionData":false},{"name":"llms-txt-v2","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false},{"name":"wildcard-email-auth","enabled":true,"variant":{"name":"disabled","enabled":false,"feature_enabled":true},"impressionData":false}],"googleFontsInlined":false},"customCss":[{"filePath":"style.css","content":"*{border-radius:0px!important}[data-theme=\"light\"] pre,[data-theme=\"light\"] pre code{background-color:#f6f8fa!important;color:#24292e!important}[data-theme=\"dark\"] pre,[data-theme=\"dark\"] pre code{background-color:#161b22!important;color:#e1e4e8!important}[data-theme=\"light\"] code:not(pre code){color:#0969da!important;background-color:#f6f8fa!important;padding:2px 4px;border-radius:3px}[data-theme=\"dark\"] code:not(pre code){color:#79c0ff!important;background-color:#161b22!important;padding:2px 4px;border-radius:3px}"}]}
{"page":{"pagePath":"api-reference/agents/evals/create-eval","slug":"/api-reference/agents/evals/create-eval","pageMetadata":{"title":"Create Eval","api":"POST /agents/evals","description":"Score a blueprint against a set of labeled examples","tag":"v1.1.12+","href":"/api-reference/agents/evals/create-eval"},"description":{"compiledSource":"\"use strict\";\nconst {jsx: _jsx} = arguments[0];\nconst {useMDXComponents: _provideComponents} = arguments[0];\nfunction _createMdxContent(props) {\n const _components = {\n p: \"p\",\n ..._provideComponents(),\n ...props.components\n };\n return _jsx(_components.p, {\n children: \"Score a blueprint against a set of labeled examples\"\n });\n}\nfunction MDXContent(props = {}) {\n const {wrapper: MDXLayout} = {\n ..._provideComponents(),\n ...props.components\n };\n return MDXLayout ? _jsx(MDXLayout, {\n ...props,\n children: _jsx(_createMdxContent, {\n ...props\n })\n }) : _createMdxContent(props);\n}\nreturn {\n default: MDXContent\n};\n","frontmatter":{},"scope":{}},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,li:`li`,p:`p`,ul:`ul`,..._provideComponents(),...props.components},{Badge,Callout,Heading,Note,ParamField,ResponseField}=_components;if(!Badge)_missingMdxReference(`Badge`,true);if(!Callout)_missingMdxReference(`Callout`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);if(!ParamField)_missingMdxReference(`ParamField`,true);if(!ResponseField)_missingMdxReference(`ResponseField`,true);return _jsxs(_Fragment,{children:[_jsx(Callout,{icon:`clock`,color:`#3064E3`,iconType:`solid`,children:_jsxs(_components.p,{children:[`Requires `,_jsx(_components.a,{href:`/release-notes/1.1.x#v1-1-12`,children:`version 1.1.12`}),` or later of the Archetype platform.`]})}),`\n`,_jsx(Heading,{level:`2`,id:`overview`,children:`Overview`}),`\n`,_jsx(_components.p,{children:`This endpoint creates and starts an evaluation: the blueprint to evaluate, the examples to score\nit on, and — only where the run differs from what the blueprint declares — how to score them.`}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[`Only blueprints that declare a metrics schema support being evaluated. The blueprint must also\nbe complete: a blueprint that declares values with no default (written `,_jsx(_components.code,{children:\"${required}\"}),`), which\nonly an optimization can provide, cannot be evaluated directly.`]})}),`\n`,_jsxs(_components.p,{children:[`Per-target scoring overrides and the headline `,_jsx(_components.code,{children:`primary`}),` metric are resolved against the\nblueprint at creation, so the returned eval records exactly what will be scored.`]}),`\n`,_jsxs(_components.p,{children:[`The eval is created `,_jsx(_components.code,{children:`pending`}),` and returns immediately; it moves to `,_jsx(_components.code,{children:`running`}),` and then to\n`,_jsx(_components.code,{children:`completed`}),`, `,_jsx(_components.code,{children:`failed`}),`, or `,_jsx(_components.code,{children:`cancelled`}),` on its own. Poll `,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval`,children:`Get\nEval`}),` to follow it and to read its `,_jsx(_components.code,{children:`metrics_report`}),` and\n`,_jsx(_components.code,{children:`output_artifacts`}),`. Use `,_jsx(_components.a,{href:`/api-reference/agents/evals/list-eval-examples`,children:`List Eval Examples`}),` to\nfind out where it scored badly.`]}),`\n`,_jsxs(_components.p,{children:[`The request is rejected with `,_jsx(_components.code,{children:`400`}),` when it contradicts the blueprint’s metrics schema, when a\nconstraint bounds an objective the run does not compute, or when the blueprint is incomplete (it\ndeclares values with no default, written `,_jsx(_components.code,{children:\"${required}\"}),`). It is rejected with `,_jsx(_components.code,{children:`403`}),`\n(`,_jsx(_components.code,{children:`capability_locked`}),`) when the blueprint is not available to the caller’s organization, and with\n`,_jsx(_components.code,{children:`404`}),` when the blueprint is not found.`]}),`\n`,_jsxs(_components.p,{children:[`The request is rejected with `,_jsx(_components.code,{children:`422`}),` when a field is malformed, including a `,_jsx(_components.code,{children:`primary`}),` or a\nconstraint `,_jsx(_components.code,{children:`objective`}),` that is not written `,_jsx(_components.code,{children:`\u003ctarget>.\u003cobjective>`}),`.`]}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[`Starting with `,_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`version 2.0.0`}),`, a malformed `,_jsx(_components.code,{children:`primary`}),` is rejected\nwith `,_jsx(_components.code,{children:`422`}),` instead of `,_jsx(_components.code,{children:`400`}),`.`]})}),`\n`,_jsxs(_components.p,{children:[`An eval of a blueprint in a `,_jsx(_components.a,{href:`/api-reference/projects/about`,children:`project`}),` is in that project. Creating\nit requires being able to modify the project: you get `,_jsx(_components.code,{children:`403`}),` when you can see the project but not\nmodify it, and `,_jsx(_components.code,{children:`404`}),` when the blueprint is in a project you cannot see. The eval’s job and output\nfiles belong to the project’s creator, whoever starts it, and every input file must be one the\nproject’s creator can see. Any other input file returns `,_jsx(_components.code,{children:`400`}),`.`]}),`\n`,_jsxs(_components.p,{children:[`A blueprint outside any project, including a canonical blueprint, can’t be evaluated directly: the\nrequest returns `,_jsx(_components.code,{children:`400`}),` with code `,_jsx(_components.code,{children:`project_required`}),`. Copy the blueprint into a project with `,_jsx(_components.a,{href:`/api-reference/agents/blueprints/copy-blueprint`,children:`Copy\nBlueprint`}),` and evaluate the copy. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-1`,children:`v2.0.1+`})})]}),`\n`,_jsxs(_components.p,{children:[`Only each example’s `,_jsx(_components.code,{children:`inputs`}),` is required. An example following the conventions its blueprint\ndeclares needs nothing else: it is named after its input file, and each target’s labels are read\nfrom a column named after the target. State a deviation once: on the run’s `,_jsx(_components.code,{children:`targets`}),` when every\nexample shares it, on the example itself when only one does.`]}),`\n`,_jsx(Heading,{level:`2`,id:`request`,children:`Request`}),`\n`,_jsx(ParamField,{body:`blueprint_id`,type:`string`,required:true,id:`param-blueprint-id`,children:_jsxs(_components.p,{children:[`The blueprint to evaluate — a `,_jsx(_components.code,{children:`blp_`}),` id. Must exist and be visible to the caller’s organization.`]})}),`\n`,_jsx(ParamField,{body:`examples`,type:`array`,required:true,id:`param-examples`,children:_jsxs(_components.p,{children:[`The examples to score. Must be non-empty. See `,_jsx(_components.a,{href:`#example-object-examples`,children:`Example object`}),`.`]})}),`\n`,_jsx(ParamField,{body:`name`,type:`string`,id:`param-name`,children:_jsx(_components.p,{children:`Optional human label. Defaults to an empty string.`})}),`\n`,_jsx(ParamField,{body:`primary`,type:`string`,id:`param-primary`,children:_jsxs(_components.p,{children:[`The run’s headline, as `,_jsx(_components.code,{children:`\u003ctarget>.\u003cobjective>`}),` — e.g. `,_jsx(_components.code,{children:`state.macro_f1`}),`. Defaults to the\nblueprint’s own. Must name a target this run scores and an objective it computes for it.`]})}),`\n`,_jsx(ParamField,{body:`emit_predictions`,type:`boolean`,default:`true`,id:`param-emit-predictions`,children:_jsxs(_components.p,{children:[`Emit the predictions NDJSON artifacts alongside the metrics report: one per target, with a row\nper scored item. The artifacts are what make a score explainable. Set `,_jsx(_components.code,{children:`false`}),` to skip them when\nthe input is large enough that they are not worth their size.`]})}),`\n`,_jsx(ParamField,{body:`constraints`,type:`array`,id:`param-constraints`,children:_jsxs(_components.p,{children:[`Bounds the run reports a verdict on, replacing those the blueprint declares. Omit it to use the\nblueprint’s; send `,_jsx(_components.code,{children:`[]`}),` for none. Each entry names an objective the run computes, with a `,_jsx(_components.code,{children:`min`}),`, a\n`,_jsx(_components.code,{children:`max`}),`, or both. See `,_jsx(_components.a,{href:`#constraint-constraints`,children:`Constraint`}),`. Missing a constraint does not fail\nthe eval: `,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval#metrics-report-metrics_report`,children:`Get Eval`}),`\nreports each constraint’s verdict and whether they all hold. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]})}),`\n`,_jsxs(ParamField,{body:`targets`,type:`object`,id:`param-targets`,children:[_jsxs(_components.p,{children:[`Per-target overrides for this run, keyed by target name. Every field of an entry is optional\nand falls back to what the blueprint’s target declares, so a run that wants nothing different\nomits this entirely — and one that does need a change states it once here rather than on every\nfile. See `,_jsx(_components.a,{href:`#target-override-targets-%3Ctarget-name%3E`,children:`Target override`}),`.`]}),_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.code,{children:`targets`}),` overrides apply to every example of the run; an example may still override them\nfor itself.`]})}),_jsxs(_components.p,{children:[`A key must name a target the blueprint declares. An entry’s `,_jsx(_components.code,{children:`objectives`}),` replaces the target’s\nown list for this run, and may include objectives the target does not declare.`]})]}),`\n`,_jsxs(Heading,{level:`3`,id:`example-object-examples`,children:[`Example object (`,_jsx(_components.code,{children:`examples[]`}),`)`]}),`\n`,_jsx(_components.p,{children:`One thing an eval scores the agent on: the inputs handed to the agent, and the ground truth\nexpected of it. An eval run covers several, and reports each separately as well as together —\nwhich is what lets a run say not just how good the agent is but where it is bad.`}),`\n`,_jsxs(ParamField,{body:`inputs`,type:`array`,required:true,id:`param-inputs`,children:[_jsxs(_components.p,{children:[`The data handed to the agent for this example, in order: `,_jsx(_components.code,{children:`file`}),` references, each file used at\nmost once in a request. Each entry is a `,_jsx(_components.a,{href:`#data-ref-examples-inputs`,children:`data ref`}),`.`]}),_jsx(Note,{children:_jsxs(_components.p,{children:[`Starting with `,_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`version 2.0.0`}),`, an example may have several `,_jsx(_components.code,{children:`file`}),` references. Earlier versions\npermitted exactly one.`]})}),_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.code,{children:`inputs`}),` is the only property required on an example. A conventionally-shaped example writes\nnothing else; it is named after its input file, and each target’s labels come from a column\nnamed after the target.`]})})]}),`\n`,_jsx(ParamField,{body:`name`,type:`string`,id:`param-name-1`,children:_jsx(_components.p,{children:`The name by which this example is identified in the run’s results. Defaults to the file name of\nits input, without the extension. Names must be unique within a request: results are keyed by\nname, and two examples sharing one cannot be told apart in a report.`})}),`\n`,_jsxs(ParamField,{body:`ground_truth`,type:`object`,id:`param-ground-truth`,children:[_jsxs(_components.p,{children:[`The annotation supplied with this example, keyed by a name of your choosing: where its values\nare and, where they are not recorded at the cadence the agent emits at, how they reduce. Each\nvalue is a `,_jsx(_components.a,{href:`#ground-truth-declaration-examples-ground_truth-%3Cname%3E`,children:`ground-truth declaration`}),`.`]}),_jsxs(_components.p,{children:[`This says nothing about which target it serves; that’s done by `,_jsx(_components.code,{children:`binding`}),`, so the same\nannotation can be written once and scored against different agents. Omit `,_jsx(_components.code,{children:`ground_truth`}),`\nentirely to use the convention: each target’s labels are a column of the input named after the\ntarget.`]})]}),`\n`,_jsx(ParamField,{body:`binding`,type:`object`,id:`param-binding`,children:_jsx(_components.p,{children:`Which targets each annotation serves, keyed by the annotation’s name; each value is an array of\ntarget names. Omit it whenever the names line up: an annotation named after a target serves\nthat target, and a single annotation serves a single target whatever either is called.`})}),`\n`,_jsx(ParamField,{body:`metadata`,type:`object`,id:`param-metadata`,children:_jsx(_components.p,{children:`Free-form tags for this example. Carried through to its results, where they are what makes a\nlong per-example table answer a question — which site, which shift, which operator.`})}),`\n`,_jsxs(Heading,{level:`3`,id:`data-ref-examples-inputs`,children:[`Data ref (`,_jsx(_components.code,{children:`examples[].inputs[]`}),`)`]}),`\n`,_jsx(ParamField,{body:`type`,type:`string`,required:true,id:`param-type`,children:_jsxs(_components.p,{children:[`Storage kind, e.g. `,_jsx(_components.code,{children:`file`}),` or `,_jsx(_components.code,{children:`dataset`}),`.`]})}),`\n`,_jsx(ParamField,{body:`id`,type:`string`,required:true,id:`param-id`,children:_jsxs(_components.p,{children:[`Identifier within that storage kind, e.g. `,_jsx(_components.code,{children:`file_abc123`}),`.`]})}),`\n`,_jsx(ParamField,{body:`format`,type:`string`,id:`param-format`,children:_jsxs(_components.p,{children:[`Optional format hint, e.g. `,_jsx(_components.code,{children:`csv`}),`.`]})}),`\n`,_jsx(ParamField,{body:`crc32c`,type:`string`,id:`param-crc32c`,children:_jsxs(_components.p,{children:[`Whole-file CRC32C checksum of the referenced bytes, base64-encoded exactly as S3 emits it. You\ndon’t need to send it: when the eval is created, it’s pinned from the checksum the data service\nstores for the file, so the eval records exactly which bytes it scored even if a filename is\nlater reused. `,_jsx(_components.code,{children:`null`}),` when the data service has no checksum for the file. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/1.1.x#v1-1-13`,children:`v1.1.13+`})})]})}),`\n`,_jsx(ParamField,{body:`metadata`,type:`object`,id:`param-metadata-1`,children:_jsxs(_components.p,{children:[`Free-form detail about this ref. Any key is carried through untouched. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/1.1.x#v1-1-13`,children:`v1.1.13+`})})]})}),`\n`,_jsxs(Heading,{level:`3`,id:`ground-truth-declaration-examples-ground_truth-\u003cname>`,children:[`Ground-truth declaration (`,_jsx(_components.code,{children:`examples[].ground_truth.\u003cname>`}),`)`]}),`\n`,_jsxs(_components.p,{children:[`One annotation supplied with an example. Both fields of this object are optional and layer over\nwhat the run already settled, because the two are independent statements: an annotation in a\ncolumn of its own overrides `,_jsx(_components.code,{children:`from`}),` and leaves the reduction alone; an annotation recorded at a\ncoarser cadence than the rest of the set overrides `,_jsx(_components.code,{children:`downsampling`}),` and leaves its location alone.`]}),`\n`,_jsxs(ParamField,{body:`from`,type:`object`,id:`param-from`,children:[_jsx(_components.p,{children:`Where this example keeps the annotation’s values. Omitted to keep whatever the run settled on —\nby default, a column named after the target it serves. One of three forms:`}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`{\"column\": \"state\"}`}),` — the labels are a column of the input itself, one label per record.\nInputs of the same request may each name a different column. An optional `,_jsx(_components.code,{children:`values`}),` map\n(`,_jsx(_components.code,{children:`{\"0\": \"idle\", \"1\": \"running\"}`}),`) maps each value as it appears in the column onto the label\nit means, for an input that spells its labels differently — integer codes, a vendor’s own\nwording. Omitted means the column already holds the labels. A value the map does not cover\nis a read error, not a silently dropped record.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`{\"constant\": \"idle\"}`}),` — one label covers the whole input, and every record of the input is\nscored against it.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`{\"per_input\": {\"fil_idle\": \"idle\", \"fil_drilling\": \"drilling\"}}`}),` — each input carries a\nlabel of its own, which covers the whole of it. Keys are the `,_jsx(_components.code,{children:`id`}),`s of the example’s `,_jsx(_components.code,{children:`inputs`}),`,\nand values are labels of the target. An input left out has no label for this target and is\nnot scored against it. Use this to group files already separated one per class into a single\nexample. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]}),`\n`]}),_jsx(_components.p,{children:`Unknown keys in either form are rejected rather than ignored.`})]}),`\n`,_jsxs(ParamField,{body:`downsampling`,type:`string`,id:`param-downsampling`,children:[_jsxs(_components.p,{children:[`For a target paired `,_jsx(_components.code,{children:`by_time`}),` only: how the values of the records making up one analysis window\nreduce to the single value that window is scored against: `,_jsx(_components.code,{children:`last_record`}),` or `,_jsx(_components.code,{children:`majority_vote`}),`.\nOmitted to keep the rule the run settled on.`]}),_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.code,{children:`downsampling`}),` is refused for a target paired any other way, which has nothing to reduce.`]})})]}),`\n`,_jsxs(Heading,{level:`3`,id:`target-override-targets-\u003ctarget-name>`,children:[`Target override (`,_jsx(_components.code,{children:`targets.\u003ctarget name>`}),`)`]}),`\n`,_jsx(_components.p,{children:`What one run wants differently about one target. Every field is optional and falls back to the\ntarget’s own declaration. The fields here are the scoring protocol — how the agent is measured —\nnever what the agent decides or where its decision lands, which only the blueprint may say.`}),`\n`,_jsxs(ParamField,{body:`objectives`,type:`array`,id:`param-objectives`,children:[_jsx(_components.p,{children:`The objectives to compute for the target in this run, replacing those the target declares. The\nlist may include objectives the target does not declare. Each entry is one of:`}),_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`the name of an objective the target declares, which keeps that declaration`}),`\n`,_jsxs(_components.li,{children:[`a catalog objective that takes no argument, named bare: `,_jsx(_components.code,{children:`macro_f1`}),`, `,_jsx(_components.code,{children:`accuracy`}),`,\n`,_jsx(_components.code,{children:`balanced_accuracy`}),`, `,_jsx(_components.code,{children:`foreground_macro_f1`}),`, `,_jsx(_components.code,{children:`detection_recall`}),`, or `,_jsx(_components.code,{children:`detection_precision`})]}),`\n`,_jsxs(_components.li,{children:[`an objective written in full, as an object with the catalog objective as its key and its\narguments as the value, plus an optional `,_jsx(_components.code,{children:`name`})]}),`\n`]}),_jsxs(_components.p,{children:[`The per-class objectives `,_jsx(_components.code,{children:`precision`}),`, `,_jsx(_components.code,{children:`recall`}),`, and `,_jsx(_components.code,{children:`f1`}),` take a required `,_jsx(_components.code,{children:`class`}),` argument, one\nof the target’s labels, so they are always written in full: for example,\n`,_jsx(_components.code,{children:`{\"name\": \"recall_cold\", \"recall\": {\"class\": \"cold\"}}`}),`. `,_jsx(_components.code,{children:`name`}),` is what `,_jsx(_components.code,{children:`primary`}),` and the metrics\nreport refer to the objective by, and defaults to the catalog objective’s name. Names must be\nunique within a target, so give each one a `,_jsx(_components.code,{children:`name`}),` to compute the same per-class objective for\nseveral classes. Written in full, an objective may reuse a name the target declares, and then\nreplaces that declaration for the run. `,_jsx(_components.code,{children:`macro_f1`}),` is shorthand for `,_jsx(_components.code,{children:`{\"macro_f1\": {}}`}),`.`]}),_jsxs(_components.p,{children:[`Written in full, an objective may also set `,_jsx(_components.code,{children:`if_undefined`}),`: the number it takes wherever it has\nno value of its own, such as an example with nothing to compute it from. For example,\n`,_jsx(_components.code,{children:`{\"mean_iou\": {}, \"if_undefined\": 1}`}),`. The value must be within the range of values the\nobjective takes, or the eval is rejected when it is created. The metrics report lists the\nobjectives that took this value in `,_jsx(_components.code,{children:`defaulted`}),`. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-1`,children:`v2.0.1+`})})]}),_jsx(Note,{children:_jsxs(_components.p,{children:[`Starting with `,_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`version 2.0.0`}),`, `,_jsx(_components.code,{children:`objectives`}),` replaces the target’s list and may add objectives the\ntarget does not declare. Earlier versions could only narrow the list.`]})}),_jsx(_components.p,{children:`The catalog is closed: an objective outside it is rejected when the eval is created, rather\nthan producing an unscored run.`})]}),`\n`,_jsx(ParamField,{body:`from`,type:`object`,id:`param-from-1`,children:_jsxs(_components.p,{children:[`Where every input of this run keeps the target’s labels, when they all agree and disagree with\nthe target’s name. An input may still override it. Same forms as a ground-truth\ndeclaration’s `,_jsx(_components.code,{children:`from`}),`.`]})}),`\n`,_jsxs(ParamField,{body:`downsampling`,type:`string`,id:`param-downsampling-1`,children:[_jsxs(_components.p,{children:[`For a target paired `,_jsx(_components.code,{children:`by_time`}),` only: how the values of one analysis window reduce to the single\nvalue it is scored against, for every example of this run: `,_jsx(_components.code,{children:`last_record`}),` or `,_jsx(_components.code,{children:`majority_vote`}),`. An\nexample may still override it.`]}),_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.code,{children:`downsampling`}),` is refused for a target paired any other way.`]})})]}),`\n`,_jsxs(Heading,{level:`3`,id:`constraint-constraints`,children:[`Constraint (`,_jsx(_components.code,{children:`constraints[]`}),`)`]}),`\n`,_jsxs(_components.p,{children:[`A bound on one objective the run computes: a floor, a ceiling, or both. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]}),`\n`,_jsx(ParamField,{body:`objective`,type:`string`,required:true,id:`param-objective`,children:_jsxs(_components.p,{children:[`The objective bounded, as `,_jsx(_components.code,{children:`\u003ctarget>.\u003cobjective>`}),`, naming an objective the run computes for that\ntarget, with `,_jsx(_components.code,{children:`targets`}),` applied. For example, `,_jsx(_components.code,{children:`state.macro_f1`}),`, or `,_jsx(_components.code,{children:`state.recall_idle`}),` for a\nper-class objective added through `,_jsx(_components.code,{children:`targets`}),`.`]})}),`\n`,_jsx(ParamField,{body:`min`,type:`number`,id:`param-min`,children:_jsx(_components.p,{children:`The lowest acceptable value, inclusive.`})}),`\n`,_jsx(ParamField,{body:`max`,type:`number`,id:`param-max`,children:_jsx(_components.p,{children:`The highest acceptable value, inclusive.`})}),`\n`,_jsx(Heading,{level:`2`,id:`response`,children:`Response`}),`\n`,_jsxs(_components.p,{children:[`Returns `,_jsx(_components.code,{children:`201 Created`}),` with the eval in `,_jsx(_components.code,{children:`pending`}),` status, in the same format\n`,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval`,children:`Get Eval`}),` returns.`]}),`\n`,_jsx(ResponseField,{name:`id`,type:`string`,required:true,id:`param-id-1`,children:_jsxs(_components.p,{children:[`The eval’s `,_jsx(_components.code,{children:`evl_`}),` id.`]})}),`\n`,_jsx(ResponseField,{name:`name`,type:`string`,required:true,id:`param-name-2`,children:_jsx(_components.p,{children:`Human label given on create. Empty when none was given.`})}),`\n`,_jsx(ResponseField,{name:`org_id`,type:`string`,required:true,id:`param-org-id`,children:_jsx(_components.p,{children:`The organization that owns the eval.`})}),`\n`,_jsx(ResponseField,{name:`blueprint_id`,type:`string`,required:true,id:`param-blueprint-id-1`,children:_jsx(_components.p,{children:`The blueprint being evaluated.`})}),`\n`,_jsx(ResponseField,{name:`primary`,type:`string`,required:true,id:`param-primary-1`,children:_jsxs(_components.p,{children:[`The run’s headline, as `,_jsx(_components.code,{children:`\u003ctarget>.\u003cobjective>`}),` — resolved at creation, so this is what was\nactually scored rather than what was asked for.`]})}),`\n`,_jsx(ResponseField,{name:`resolved_targets`,type:`object`,required:true,id:`param-resolved-targets`,children:_jsxs(_components.p,{children:[`The objectives this run computes, keyed by target name — the blueprint’s own declaration folded\ntogether with the request’s per-target overrides, resolved at creation. Each objective is in the\nsame form as in a `,_jsx(_components.a,{href:`#target-override-targets-%3Ctarget-name%3E`,children:`target override`}),`: a bare name, or\nwritten in full with its arguments. `,_jsx(_components.code,{children:`primary`}),` names one of these, as `,_jsx(_components.code,{children:`\u003ctarget>.\u003cname>`}),`. Read this\nrather than the blueprint’s `,_jsx(_components.code,{children:`metrics`}),` to know what was scored, since a request may replace a\ntarget’s objectives.`]})}),`\n`,_jsx(ResponseField,{name:`constraints`,type:`array`,required:true,id:`param-constraints-1`,children:_jsxs(_components.p,{children:[`The bounds the run reports a verdict on, resolved at creation: the request’s, or else the\nblueprint’s. Empty when there are none. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]})}),`\n`,_jsx(ResponseField,{name:`emit_predictions`,type:`boolean`,required:true,id:`param-emit-predictions-1`,children:_jsx(_components.p,{children:`Whether the run emits the predictions artifacts alongside the metrics report, one per target,\nas the request asked.`})}),`\n`,_jsx(ResponseField,{name:`examples`,type:`array`,required:true,id:`param-examples-1`,children:_jsxs(_components.p,{children:[`The examples the eval was created against, as resolved: each carries the name its results are\nkeyed by, and inputs stamped with the CRC32C of the bytes scored. See `,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval#resolved-example-object-examples`,children:`Resolved example\nobject`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`status`,type:`string`,required:true,id:`param-status`,children:_jsxs(_components.p,{children:[`The eval’s current lifecycle status; `,_jsx(_components.code,{children:`pending`}),` on creation.`]})}),`\n`,_jsx(ResponseField,{name:`project_id`,type:`string`,id:`param-project-id`,children:_jsxs(_components.p,{children:[`The `,_jsx(_components.a,{href:`/api-reference/projects/about`,children:`project`}),` the eval is in: its blueprint’s. This value is `,_jsx(_components.code,{children:`null`}),` outside any\nproject. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]})}),`\n`,_jsx(ResponseField,{name:`created_by`,type:`string`,required:true,id:`param-created-by`,children:_jsxs(_components.p,{children:[`Subject id (`,_jsx(_components.code,{children:`usr_...`}),` or `,_jsx(_components.code,{children:`key_...`}),`) that created this eval.`]})}),`\n`,_jsx(ResponseField,{name:`created_at`,type:`string`,required:true,id:`param-created-at`,children:_jsx(_components.p,{children:`When the eval was created (date-time).`})}),`\n`,_jsx(ResponseField,{name:`started_at`,type:`string`,id:`param-started-at`,children:_jsxs(_components.p,{children:[`When scoring began. Absent while `,_jsx(_components.code,{children:`pending`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`completed_at`,type:`string`,id:`param-completed-at`,children:_jsx(_components.p,{children:`When the eval reached a terminal status. Absent until then.`})}),`\n`,_jsx(ResponseField,{name:`metrics_report`,type:`object`,id:`param-metrics-report`,children:_jsxs(_components.p,{children:[`What the run measured. Present once the eval is `,_jsx(_components.code,{children:`completed`}),`. See\n`,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval#metrics-report-metrics_report`,children:`Get Eval`}),` for its shape.`]})}),`\n`,_jsx(ResponseField,{name:`output_artifacts`,type:`array`,id:`param-output-artifacts`,children:_jsxs(_components.p,{children:[`Refs to the files the run produced, filled in while the eval runs. Each `,_jsx(_components.code,{children:`id`}),` is a data-service\nfile id, so an artifact is downloadable through the files API.`]})}),`\n`,_jsx(ResponseField,{name:`error`,type:`string`,id:`param-error`,children:_jsxs(_components.p,{children:[`Failure detail; `,_jsx(_components.code,{children:`null`}),` unless the eval failed.`]})}),`\n`,_jsxs(Heading,{level:`3`,id:`resolved-example-object-examples`,children:[`Resolved example object (`,_jsx(_components.code,{children:`examples[]`}),`)`]}),`\n`,_jsxs(ResponseField,{name:`name`,type:`string`,required:true,id:`param-name-3`,children:[_jsx(_components.p,{children:`How this example is identified in the run’s results.`}),_jsx(Note,{children:_jsx(_components.p,{children:`Example names must be unique within a request, because results are keyed by name.`})})]}),`\n`,_jsx(ResponseField,{name:`ordinal`,type:`integer`,required:true,id:`param-ordinal`,children:_jsx(_components.p,{children:`The example’s 1-based position in the run, fixing the order results are listed in.`})}),`\n`,_jsx(ResponseField,{name:`inputs`,type:`array`,required:true,id:`param-inputs-1`,children:_jsxs(_components.p,{children:[`The data handed to the agent, each ref carrying the CRC32C of the bytes scored. See `,_jsx(_components.a,{href:`#resolved-input-ref-examples-inputs`,children:`Resolved\ninput ref`}),` for the keys the service writes into each ref’s\n`,_jsx(_components.code,{children:`metadata`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`metadata`,type:`object`,id:`param-metadata-2`,children:_jsx(_components.p,{children:`The free-form tags the example was created with, if any.`})}),`\n`,_jsxs(Heading,{level:`3`,id:`resolved-input-ref-examples-inputs`,children:[`Resolved input ref (`,_jsx(_components.code,{children:`examples[].inputs[]`}),`)`]}),`\n`,_jsxs(_components.p,{children:[`Each input of a returned example is a data ref, with the same `,_jsx(_components.code,{children:`type`}),`, `,_jsx(_components.code,{children:`id`}),`, and `,_jsx(_components.code,{children:`format`}),` as in the\nrequest, plus the `,_jsx(_components.code,{children:`crc32c`}),` of the bytes scored. Its `,_jsx(_components.code,{children:`metadata`}),` carries the following keys, written\nby the service when the eval is created, never by the caller. Any other key is carried through\nuntouched.`]}),`\n`,_jsx(ResponseField,{name:`metadata.targets`,type:`object`,id:`param-metadata-targets`,children:_jsxs(_components.p,{children:[`Where this input’s ground truth is for every target the eval scores, with nothing left to\nconvention: a map of target name to that target’s `,_jsx(_components.a,{href:`#resolved-ground-truth-examples-inputs-metadata-targets-%3Ctarget-name%3E`,children:`resolved ground\ntruth`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`metadata.example_name`,type:`string`,id:`param-metadata-example-name`,children:_jsxs(_components.p,{children:[`The name of the example the input belongs to. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/1.1.x#v1-1-13`,children:`v1.1.13+`})})]})}),`\n`,_jsxs(Heading,{level:`3`,id:`resolved-ground-truth-examples-inputs-metadata-targets-\u003ctarget-name>`,children:[`Resolved ground truth (`,_jsx(_components.code,{children:`examples[].inputs[].metadata.targets.\u003ctarget name>`}),`)`]}),`\n`,_jsxs(_components.p,{children:[`A ground truth with nothing left to convention: where its values are, and how they reduce. This is\nwhat an example’s `,_jsx(_components.a,{href:`#ground-truth-declaration-examples-ground_truth-%3Cname%3E`,children:`ground-truth declaration`}),`\nbecomes once laid over the defaults the run settled on.`]}),`\n`,_jsx(ResponseField,{name:`from`,type:`object`,required:true,id:`param-from-2`,children:_jsxs(_components.p,{children:[`Where the values are. Same forms as a ground-truth declaration’s `,_jsx(_components.code,{children:`from`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`downsampling`,type:`string`,id:`param-downsampling-2`,children:_jsxs(_components.p,{children:[`For a target paired `,_jsx(_components.code,{children:`by_time`}),`, how the values of the records making up one analysis window\nreduce to the single value that window is scored against: the target’s own rule, unless the run\nor the example overrides it. Absent for a target paired any other way, which has nothing to\nreduce.`]})})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"codeStyling":"system","config":{},"pageMetadata":{"title":"Create Eval","api":"POST /agents/evals","description":"Score a blueprint against a set of labeled examples","tag":"v1.1.12+","href":"/api-reference/agents/evals/create-eval"}}},"mdxSourceWithNoJs":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];function _createMdxContent(props){const _components={a:`a`,code:`code`,li:`li`,p:`p`,pre:`pre`,span:`span`,ul:`ul`,..._provideComponents(),...props.components},{Badge,Callout,CodeBlock,Heading,Note,ParamField,RequestExample,ResponseExample,ResponseField}=_components;if(!Badge)_missingMdxReference(`Badge`,true);if(!Callout)_missingMdxReference(`Callout`,true);if(!CodeBlock)_missingMdxReference(`CodeBlock`,true);if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);if(!ParamField)_missingMdxReference(`ParamField`,true);if(!RequestExample)_missingMdxReference(`RequestExample`,true);if(!ResponseExample)_missingMdxReference(`ResponseExample`,true);if(!ResponseField)_missingMdxReference(`ResponseField`,true);return _jsxs(_Fragment,{children:[_jsx(Callout,{icon:`clock`,color:`#3064E3`,iconType:`solid`,children:_jsxs(_components.p,{children:[`Requires `,_jsx(_components.a,{href:`/release-notes/1.1.x#v1-1-12`,children:`version 1.1.12`}),` or later of the Archetype platform.`]})}),`\n`,_jsx(Heading,{level:`2`,id:`overview`,children:`Overview`}),`\n`,_jsx(_components.p,{children:`This endpoint creates and starts an evaluation: the blueprint to evaluate, the examples to score\nit on, and — only where the run differs from what the blueprint declares — how to score them.`}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[`Only blueprints that declare a metrics schema support being evaluated. The blueprint must also\nbe complete: a blueprint that declares values with no default (written `,_jsx(_components.code,{children:\"${required}\"}),`), which\nonly an optimization can provide, cannot be evaluated directly.`]})}),`\n`,_jsxs(_components.p,{children:[`Per-target scoring overrides and the headline `,_jsx(_components.code,{children:`primary`}),` metric are resolved against the\nblueprint at creation, so the returned eval records exactly what will be scored.`]}),`\n`,_jsxs(_components.p,{children:[`The eval is created `,_jsx(_components.code,{children:`pending`}),` and returns immediately; it moves to `,_jsx(_components.code,{children:`running`}),` and then to\n`,_jsx(_components.code,{children:`completed`}),`, `,_jsx(_components.code,{children:`failed`}),`, or `,_jsx(_components.code,{children:`cancelled`}),` on its own. Poll `,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval`,children:`Get\nEval`}),` to follow it and to read its `,_jsx(_components.code,{children:`metrics_report`}),` and\n`,_jsx(_components.code,{children:`output_artifacts`}),`. Use `,_jsx(_components.a,{href:`/api-reference/agents/evals/list-eval-examples`,children:`List Eval Examples`}),` to\nfind out where it scored badly.`]}),`\n`,_jsxs(_components.p,{children:[`The request is rejected with `,_jsx(_components.code,{children:`400`}),` when it contradicts the blueprint’s metrics schema, when a\nconstraint bounds an objective the run does not compute, or when the blueprint is incomplete (it\ndeclares values with no default, written `,_jsx(_components.code,{children:\"${required}\"}),`). It is rejected with `,_jsx(_components.code,{children:`403`}),`\n(`,_jsx(_components.code,{children:`capability_locked`}),`) when the blueprint is not available to the caller’s organization, and with\n`,_jsx(_components.code,{children:`404`}),` when the blueprint is not found.`]}),`\n`,_jsxs(_components.p,{children:[`The request is rejected with `,_jsx(_components.code,{children:`422`}),` when a field is malformed, including a `,_jsx(_components.code,{children:`primary`}),` or a\nconstraint `,_jsx(_components.code,{children:`objective`}),` that is not written `,_jsx(_components.code,{children:`\u003ctarget>.\u003cobjective>`}),`.`]}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[`Starting with `,_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`version 2.0.0`}),`, a malformed `,_jsx(_components.code,{children:`primary`}),` is rejected\nwith `,_jsx(_components.code,{children:`422`}),` instead of `,_jsx(_components.code,{children:`400`}),`.`]})}),`\n`,_jsxs(_components.p,{children:[`An eval of a blueprint in a `,_jsx(_components.a,{href:`/api-reference/projects/about`,children:`project`}),` is in that project. Creating\nit requires being able to modify the project: you get `,_jsx(_components.code,{children:`403`}),` when you can see the project but not\nmodify it, and `,_jsx(_components.code,{children:`404`}),` when the blueprint is in a project you cannot see. The eval’s job and output\nfiles belong to the project’s creator, whoever starts it, and every input file must be one the\nproject’s creator can see. Any other input file returns `,_jsx(_components.code,{children:`400`}),`.`]}),`\n`,_jsxs(_components.p,{children:[`A blueprint outside any project, including a canonical blueprint, can’t be evaluated directly: the\nrequest returns `,_jsx(_components.code,{children:`400`}),` with code `,_jsx(_components.code,{children:`project_required`}),`. Copy the blueprint into a project with `,_jsx(_components.a,{href:`/api-reference/agents/blueprints/copy-blueprint`,children:`Copy\nBlueprint`}),` and evaluate the copy. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-1`,children:`v2.0.1+`})})]}),`\n`,_jsxs(_components.p,{children:[`Only each example’s `,_jsx(_components.code,{children:`inputs`}),` is required. An example following the conventions its blueprint\ndeclares needs nothing else: it is named after its input file, and each target’s labels are read\nfrom a column named after the target. State a deviation once: on the run’s `,_jsx(_components.code,{children:`targets`}),` when every\nexample shares it, on the example itself when only one does.`]}),`\n`,_jsx(Heading,{level:`2`,id:`request`,children:`Request`}),`\n`,_jsx(ParamField,{body:`blueprint_id`,type:`string`,required:true,id:`param-blueprint-id`,children:_jsxs(_components.p,{children:[`The blueprint to evaluate — a `,_jsx(_components.code,{children:`blp_`}),` id. Must exist and be visible to the caller’s organization.`]})}),`\n`,_jsx(ParamField,{body:`examples`,type:`array`,required:true,id:`param-examples`,children:_jsxs(_components.p,{children:[`The examples to score. Must be non-empty. See `,_jsx(_components.a,{href:`#example-object-examples`,children:`Example object`}),`.`]})}),`\n`,_jsx(ParamField,{body:`name`,type:`string`,id:`param-name`,children:_jsx(_components.p,{children:`Optional human label. Defaults to an empty string.`})}),`\n`,_jsx(ParamField,{body:`primary`,type:`string`,id:`param-primary`,children:_jsxs(_components.p,{children:[`The run’s headline, as `,_jsx(_components.code,{children:`\u003ctarget>.\u003cobjective>`}),` — e.g. `,_jsx(_components.code,{children:`state.macro_f1`}),`. Defaults to the\nblueprint’s own. Must name a target this run scores and an objective it computes for it.`]})}),`\n`,_jsx(ParamField,{body:`emit_predictions`,type:`boolean`,default:`true`,id:`param-emit-predictions`,children:_jsxs(_components.p,{children:[`Emit the predictions NDJSON artifacts alongside the metrics report: one per target, with a row\nper scored item. The artifacts are what make a score explainable. Set `,_jsx(_components.code,{children:`false`}),` to skip them when\nthe input is large enough that they are not worth their size.`]})}),`\n`,_jsx(ParamField,{body:`constraints`,type:`array`,id:`param-constraints`,children:_jsxs(_components.p,{children:[`Bounds the run reports a verdict on, replacing those the blueprint declares. Omit it to use the\nblueprint’s; send `,_jsx(_components.code,{children:`[]`}),` for none. Each entry names an objective the run computes, with a `,_jsx(_components.code,{children:`min`}),`, a\n`,_jsx(_components.code,{children:`max`}),`, or both. See `,_jsx(_components.a,{href:`#constraint-constraints`,children:`Constraint`}),`. Missing a constraint does not fail\nthe eval: `,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval#metrics-report-metrics_report`,children:`Get Eval`}),`\nreports each constraint’s verdict and whether they all hold. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]})}),`\n`,_jsxs(ParamField,{body:`targets`,type:`object`,id:`param-targets`,children:[_jsxs(_components.p,{children:[`Per-target overrides for this run, keyed by target name. Every field of an entry is optional\nand falls back to what the blueprint’s target declares, so a run that wants nothing different\nomits this entirely — and one that does need a change states it once here rather than on every\nfile. See `,_jsx(_components.a,{href:`#target-override-targets-%3Ctarget-name%3E`,children:`Target override`}),`.`]}),_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.code,{children:`targets`}),` overrides apply to every example of the run; an example may still override them\nfor itself.`]})}),_jsxs(_components.p,{children:[`A key must name a target the blueprint declares. An entry’s `,_jsx(_components.code,{children:`objectives`}),` replaces the target’s\nown list for this run, and may include objectives the target does not declare.`]})]}),`\n`,_jsxs(Heading,{level:`3`,id:`example-object-examples`,children:[`Example object (`,_jsx(_components.code,{children:`examples[]`}),`)`]}),`\n`,_jsx(_components.p,{children:`One thing an eval scores the agent on: the inputs handed to the agent, and the ground truth\nexpected of it. An eval run covers several, and reports each separately as well as together —\nwhich is what lets a run say not just how good the agent is but where it is bad.`}),`\n`,_jsxs(ParamField,{body:`inputs`,type:`array`,required:true,id:`param-inputs`,children:[_jsxs(_components.p,{children:[`The data handed to the agent for this example, in order: `,_jsx(_components.code,{children:`file`}),` references, each file used at\nmost once in a request. Each entry is a `,_jsx(_components.a,{href:`#data-ref-examples-inputs`,children:`data ref`}),`.`]}),_jsx(Note,{children:_jsxs(_components.p,{children:[`Starting with `,_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`version 2.0.0`}),`, an example may have several `,_jsx(_components.code,{children:`file`}),` references. Earlier versions\npermitted exactly one.`]})}),_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.code,{children:`inputs`}),` is the only property required on an example. A conventionally-shaped example writes\nnothing else; it is named after its input file, and each target’s labels come from a column\nnamed after the target.`]})})]}),`\n`,_jsx(ParamField,{body:`name`,type:`string`,id:`param-name-1`,children:_jsx(_components.p,{children:`The name by which this example is identified in the run’s results. Defaults to the file name of\nits input, without the extension. Names must be unique within a request: results are keyed by\nname, and two examples sharing one cannot be told apart in a report.`})}),`\n`,_jsxs(ParamField,{body:`ground_truth`,type:`object`,id:`param-ground-truth`,children:[_jsxs(_components.p,{children:[`The annotation supplied with this example, keyed by a name of your choosing: where its values\nare and, where they are not recorded at the cadence the agent emits at, how they reduce. Each\nvalue is a `,_jsx(_components.a,{href:`#ground-truth-declaration-examples-ground_truth-%3Cname%3E`,children:`ground-truth declaration`}),`.`]}),_jsxs(_components.p,{children:[`This says nothing about which target it serves; that’s done by `,_jsx(_components.code,{children:`binding`}),`, so the same\nannotation can be written once and scored against different agents. Omit `,_jsx(_components.code,{children:`ground_truth`}),`\nentirely to use the convention: each target’s labels are a column of the input named after the\ntarget.`]})]}),`\n`,_jsx(ParamField,{body:`binding`,type:`object`,id:`param-binding`,children:_jsx(_components.p,{children:`Which targets each annotation serves, keyed by the annotation’s name; each value is an array of\ntarget names. Omit it whenever the names line up: an annotation named after a target serves\nthat target, and a single annotation serves a single target whatever either is called.`})}),`\n`,_jsx(ParamField,{body:`metadata`,type:`object`,id:`param-metadata`,children:_jsx(_components.p,{children:`Free-form tags for this example. Carried through to its results, where they are what makes a\nlong per-example table answer a question — which site, which shift, which operator.`})}),`\n`,_jsxs(Heading,{level:`3`,id:`data-ref-examples-inputs`,children:[`Data ref (`,_jsx(_components.code,{children:`examples[].inputs[]`}),`)`]}),`\n`,_jsx(ParamField,{body:`type`,type:`string`,required:true,id:`param-type`,children:_jsxs(_components.p,{children:[`Storage kind, e.g. `,_jsx(_components.code,{children:`file`}),` or `,_jsx(_components.code,{children:`dataset`}),`.`]})}),`\n`,_jsx(ParamField,{body:`id`,type:`string`,required:true,id:`param-id`,children:_jsxs(_components.p,{children:[`Identifier within that storage kind, e.g. `,_jsx(_components.code,{children:`file_abc123`}),`.`]})}),`\n`,_jsx(ParamField,{body:`format`,type:`string`,id:`param-format`,children:_jsxs(_components.p,{children:[`Optional format hint, e.g. `,_jsx(_components.code,{children:`csv`}),`.`]})}),`\n`,_jsx(ParamField,{body:`crc32c`,type:`string`,id:`param-crc32c`,children:_jsxs(_components.p,{children:[`Whole-file CRC32C checksum of the referenced bytes, base64-encoded exactly as S3 emits it. You\ndon’t need to send it: when the eval is created, it’s pinned from the checksum the data service\nstores for the file, so the eval records exactly which bytes it scored even if a filename is\nlater reused. `,_jsx(_components.code,{children:`null`}),` when the data service has no checksum for the file. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/1.1.x#v1-1-13`,children:`v1.1.13+`})})]})}),`\n`,_jsx(ParamField,{body:`metadata`,type:`object`,id:`param-metadata-1`,children:_jsxs(_components.p,{children:[`Free-form detail about this ref. Any key is carried through untouched. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/1.1.x#v1-1-13`,children:`v1.1.13+`})})]})}),`\n`,_jsxs(Heading,{level:`3`,id:`ground-truth-declaration-examples-ground_truth-\u003cname>`,children:[`Ground-truth declaration (`,_jsx(_components.code,{children:`examples[].ground_truth.\u003cname>`}),`)`]}),`\n`,_jsxs(_components.p,{children:[`One annotation supplied with an example. Both fields of this object are optional and layer over\nwhat the run already settled, because the two are independent statements: an annotation in a\ncolumn of its own overrides `,_jsx(_components.code,{children:`from`}),` and leaves the reduction alone; an annotation recorded at a\ncoarser cadence than the rest of the set overrides `,_jsx(_components.code,{children:`downsampling`}),` and leaves its location alone.`]}),`\n`,_jsxs(ParamField,{body:`from`,type:`object`,id:`param-from`,children:[_jsx(_components.p,{children:`Where this example keeps the annotation’s values. Omitted to keep whatever the run settled on —\nby default, a column named after the target it serves. One of three forms:`}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`{\"column\": \"state\"}`}),` — the labels are a column of the input itself, one label per record.\nInputs of the same request may each name a different column. An optional `,_jsx(_components.code,{children:`values`}),` map\n(`,_jsx(_components.code,{children:`{\"0\": \"idle\", \"1\": \"running\"}`}),`) maps each value as it appears in the column onto the label\nit means, for an input that spells its labels differently — integer codes, a vendor’s own\nwording. Omitted means the column already holds the labels. A value the map does not cover\nis a read error, not a silently dropped record.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`{\"constant\": \"idle\"}`}),` — one label covers the whole input, and every record of the input is\nscored against it.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.code,{children:`{\"per_input\": {\"fil_idle\": \"idle\", \"fil_drilling\": \"drilling\"}}`}),` — each input carries a\nlabel of its own, which covers the whole of it. Keys are the `,_jsx(_components.code,{children:`id`}),`s of the example’s `,_jsx(_components.code,{children:`inputs`}),`,\nand values are labels of the target. An input left out has no label for this target and is\nnot scored against it. Use this to group files already separated one per class into a single\nexample. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]}),`\n`]}),_jsx(_components.p,{children:`Unknown keys in either form are rejected rather than ignored.`})]}),`\n`,_jsxs(ParamField,{body:`downsampling`,type:`string`,id:`param-downsampling`,children:[_jsxs(_components.p,{children:[`For a target paired `,_jsx(_components.code,{children:`by_time`}),` only: how the values of the records making up one analysis window\nreduce to the single value that window is scored against: `,_jsx(_components.code,{children:`last_record`}),` or `,_jsx(_components.code,{children:`majority_vote`}),`.\nOmitted to keep the rule the run settled on.`]}),_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.code,{children:`downsampling`}),` is refused for a target paired any other way, which has nothing to reduce.`]})})]}),`\n`,_jsxs(Heading,{level:`3`,id:`target-override-targets-\u003ctarget-name>`,children:[`Target override (`,_jsx(_components.code,{children:`targets.\u003ctarget name>`}),`)`]}),`\n`,_jsx(_components.p,{children:`What one run wants differently about one target. Every field is optional and falls back to the\ntarget’s own declaration. The fields here are the scoring protocol — how the agent is measured —\nnever what the agent decides or where its decision lands, which only the blueprint may say.`}),`\n`,_jsxs(ParamField,{body:`objectives`,type:`array`,id:`param-objectives`,children:[_jsx(_components.p,{children:`The objectives to compute for the target in this run, replacing those the target declares. The\nlist may include objectives the target does not declare. Each entry is one of:`}),_jsxs(_components.ul,{children:[`\n`,_jsx(_components.li,{children:`the name of an objective the target declares, which keeps that declaration`}),`\n`,_jsxs(_components.li,{children:[`a catalog objective that takes no argument, named bare: `,_jsx(_components.code,{children:`macro_f1`}),`, `,_jsx(_components.code,{children:`accuracy`}),`,\n`,_jsx(_components.code,{children:`balanced_accuracy`}),`, `,_jsx(_components.code,{children:`foreground_macro_f1`}),`, `,_jsx(_components.code,{children:`detection_recall`}),`, or `,_jsx(_components.code,{children:`detection_precision`})]}),`\n`,_jsxs(_components.li,{children:[`an objective written in full, as an object with the catalog objective as its key and its\narguments as the value, plus an optional `,_jsx(_components.code,{children:`name`})]}),`\n`]}),_jsxs(_components.p,{children:[`The per-class objectives `,_jsx(_components.code,{children:`precision`}),`, `,_jsx(_components.code,{children:`recall`}),`, and `,_jsx(_components.code,{children:`f1`}),` take a required `,_jsx(_components.code,{children:`class`}),` argument, one\nof the target’s labels, so they are always written in full: for example,\n`,_jsx(_components.code,{children:`{\"name\": \"recall_cold\", \"recall\": {\"class\": \"cold\"}}`}),`. `,_jsx(_components.code,{children:`name`}),` is what `,_jsx(_components.code,{children:`primary`}),` and the metrics\nreport refer to the objective by, and defaults to the catalog objective’s name. Names must be\nunique within a target, so give each one a `,_jsx(_components.code,{children:`name`}),` to compute the same per-class objective for\nseveral classes. Written in full, an objective may reuse a name the target declares, and then\nreplaces that declaration for the run. `,_jsx(_components.code,{children:`macro_f1`}),` is shorthand for `,_jsx(_components.code,{children:`{\"macro_f1\": {}}`}),`.`]}),_jsxs(_components.p,{children:[`Written in full, an objective may also set `,_jsx(_components.code,{children:`if_undefined`}),`: the number it takes wherever it has\nno value of its own, such as an example with nothing to compute it from. For example,\n`,_jsx(_components.code,{children:`{\"mean_iou\": {}, \"if_undefined\": 1}`}),`. The value must be within the range of values the\nobjective takes, or the eval is rejected when it is created. The metrics report lists the\nobjectives that took this value in `,_jsx(_components.code,{children:`defaulted`}),`. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-1`,children:`v2.0.1+`})})]}),_jsx(Note,{children:_jsxs(_components.p,{children:[`Starting with `,_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`version 2.0.0`}),`, `,_jsx(_components.code,{children:`objectives`}),` replaces the target’s list and may add objectives the\ntarget does not declare. Earlier versions could only narrow the list.`]})}),_jsx(_components.p,{children:`The catalog is closed: an objective outside it is rejected when the eval is created, rather\nthan producing an unscored run.`})]}),`\n`,_jsx(ParamField,{body:`from`,type:`object`,id:`param-from-1`,children:_jsxs(_components.p,{children:[`Where every input of this run keeps the target’s labels, when they all agree and disagree with\nthe target’s name. An input may still override it. Same forms as a ground-truth\ndeclaration’s `,_jsx(_components.code,{children:`from`}),`.`]})}),`\n`,_jsxs(ParamField,{body:`downsampling`,type:`string`,id:`param-downsampling-1`,children:[_jsxs(_components.p,{children:[`For a target paired `,_jsx(_components.code,{children:`by_time`}),` only: how the values of one analysis window reduce to the single\nvalue it is scored against, for every example of this run: `,_jsx(_components.code,{children:`last_record`}),` or `,_jsx(_components.code,{children:`majority_vote`}),`. An\nexample may still override it.`]}),_jsx(Note,{children:_jsxs(_components.p,{children:[_jsx(_components.code,{children:`downsampling`}),` is refused for a target paired any other way.`]})})]}),`\n`,_jsxs(Heading,{level:`3`,id:`constraint-constraints`,children:[`Constraint (`,_jsx(_components.code,{children:`constraints[]`}),`)`]}),`\n`,_jsxs(_components.p,{children:[`A bound on one objective the run computes: a floor, a ceiling, or both. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]}),`\n`,_jsx(ParamField,{body:`objective`,type:`string`,required:true,id:`param-objective`,children:_jsxs(_components.p,{children:[`The objective bounded, as `,_jsx(_components.code,{children:`\u003ctarget>.\u003cobjective>`}),`, naming an objective the run computes for that\ntarget, with `,_jsx(_components.code,{children:`targets`}),` applied. For example, `,_jsx(_components.code,{children:`state.macro_f1`}),`, or `,_jsx(_components.code,{children:`state.recall_idle`}),` for a\nper-class objective added through `,_jsx(_components.code,{children:`targets`}),`.`]})}),`\n`,_jsx(ParamField,{body:`min`,type:`number`,id:`param-min`,children:_jsx(_components.p,{children:`The lowest acceptable value, inclusive.`})}),`\n`,_jsx(ParamField,{body:`max`,type:`number`,id:`param-max`,children:_jsx(_components.p,{children:`The highest acceptable value, inclusive.`})}),`\n`,_jsx(Heading,{level:`2`,id:`response`,children:`Response`}),`\n`,_jsxs(_components.p,{children:[`Returns `,_jsx(_components.code,{children:`201 Created`}),` with the eval in `,_jsx(_components.code,{children:`pending`}),` status, in the same format\n`,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval`,children:`Get Eval`}),` returns.`]}),`\n`,_jsx(ResponseField,{name:`id`,type:`string`,required:true,id:`param-id-1`,children:_jsxs(_components.p,{children:[`The eval’s `,_jsx(_components.code,{children:`evl_`}),` id.`]})}),`\n`,_jsx(ResponseField,{name:`name`,type:`string`,required:true,id:`param-name-2`,children:_jsx(_components.p,{children:`Human label given on create. Empty when none was given.`})}),`\n`,_jsx(ResponseField,{name:`org_id`,type:`string`,required:true,id:`param-org-id`,children:_jsx(_components.p,{children:`The organization that owns the eval.`})}),`\n`,_jsx(ResponseField,{name:`blueprint_id`,type:`string`,required:true,id:`param-blueprint-id-1`,children:_jsx(_components.p,{children:`The blueprint being evaluated.`})}),`\n`,_jsx(ResponseField,{name:`primary`,type:`string`,required:true,id:`param-primary-1`,children:_jsxs(_components.p,{children:[`The run’s headline, as `,_jsx(_components.code,{children:`\u003ctarget>.\u003cobjective>`}),` — resolved at creation, so this is what was\nactually scored rather than what was asked for.`]})}),`\n`,_jsx(ResponseField,{name:`resolved_targets`,type:`object`,required:true,id:`param-resolved-targets`,children:_jsxs(_components.p,{children:[`The objectives this run computes, keyed by target name — the blueprint’s own declaration folded\ntogether with the request’s per-target overrides, resolved at creation. Each objective is in the\nsame form as in a `,_jsx(_components.a,{href:`#target-override-targets-%3Ctarget-name%3E`,children:`target override`}),`: a bare name, or\nwritten in full with its arguments. `,_jsx(_components.code,{children:`primary`}),` names one of these, as `,_jsx(_components.code,{children:`\u003ctarget>.\u003cname>`}),`. Read this\nrather than the blueprint’s `,_jsx(_components.code,{children:`metrics`}),` to know what was scored, since a request may replace a\ntarget’s objectives.`]})}),`\n`,_jsx(ResponseField,{name:`constraints`,type:`array`,required:true,id:`param-constraints-1`,children:_jsxs(_components.p,{children:[`The bounds the run reports a verdict on, resolved at creation: the request’s, or else the\nblueprint’s. Empty when there are none. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]})}),`\n`,_jsx(ResponseField,{name:`emit_predictions`,type:`boolean`,required:true,id:`param-emit-predictions-1`,children:_jsx(_components.p,{children:`Whether the run emits the predictions artifacts alongside the metrics report, one per target,\nas the request asked.`})}),`\n`,_jsx(ResponseField,{name:`examples`,type:`array`,required:true,id:`param-examples-1`,children:_jsxs(_components.p,{children:[`The examples the eval was created against, as resolved: each carries the name its results are\nkeyed by, and inputs stamped with the CRC32C of the bytes scored. See `,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval#resolved-example-object-examples`,children:`Resolved example\nobject`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`status`,type:`string`,required:true,id:`param-status`,children:_jsxs(_components.p,{children:[`The eval’s current lifecycle status; `,_jsx(_components.code,{children:`pending`}),` on creation.`]})}),`\n`,_jsx(ResponseField,{name:`project_id`,type:`string`,id:`param-project-id`,children:_jsxs(_components.p,{children:[`The `,_jsx(_components.a,{href:`/api-reference/projects/about`,children:`project`}),` the eval is in: its blueprint’s. This value is `,_jsx(_components.code,{children:`null`}),` outside any\nproject. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/2.0.x#v2-0-0`,children:`v2.0.0+`})})]})}),`\n`,_jsx(ResponseField,{name:`created_by`,type:`string`,required:true,id:`param-created-by`,children:_jsxs(_components.p,{children:[`Subject id (`,_jsx(_components.code,{children:`usr_...`}),` or `,_jsx(_components.code,{children:`key_...`}),`) that created this eval.`]})}),`\n`,_jsx(ResponseField,{name:`created_at`,type:`string`,required:true,id:`param-created-at`,children:_jsx(_components.p,{children:`When the eval was created (date-time).`})}),`\n`,_jsx(ResponseField,{name:`started_at`,type:`string`,id:`param-started-at`,children:_jsxs(_components.p,{children:[`When scoring began. Absent while `,_jsx(_components.code,{children:`pending`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`completed_at`,type:`string`,id:`param-completed-at`,children:_jsx(_components.p,{children:`When the eval reached a terminal status. Absent until then.`})}),`\n`,_jsx(ResponseField,{name:`metrics_report`,type:`object`,id:`param-metrics-report`,children:_jsxs(_components.p,{children:[`What the run measured. Present once the eval is `,_jsx(_components.code,{children:`completed`}),`. See\n`,_jsx(_components.a,{href:`/api-reference/agents/evals/get-eval#metrics-report-metrics_report`,children:`Get Eval`}),` for its shape.`]})}),`\n`,_jsx(ResponseField,{name:`output_artifacts`,type:`array`,id:`param-output-artifacts`,children:_jsxs(_components.p,{children:[`Refs to the files the run produced, filled in while the eval runs. Each `,_jsx(_components.code,{children:`id`}),` is a data-service\nfile id, so an artifact is downloadable through the files API.`]})}),`\n`,_jsx(ResponseField,{name:`error`,type:`string`,id:`param-error`,children:_jsxs(_components.p,{children:[`Failure detail; `,_jsx(_components.code,{children:`null`}),` unless the eval failed.`]})}),`\n`,_jsxs(Heading,{level:`3`,id:`resolved-example-object-examples`,children:[`Resolved example object (`,_jsx(_components.code,{children:`examples[]`}),`)`]}),`\n`,_jsxs(ResponseField,{name:`name`,type:`string`,required:true,id:`param-name-3`,children:[_jsx(_components.p,{children:`How this example is identified in the run’s results.`}),_jsx(Note,{children:_jsx(_components.p,{children:`Example names must be unique within a request, because results are keyed by name.`})})]}),`\n`,_jsx(ResponseField,{name:`ordinal`,type:`integer`,required:true,id:`param-ordinal`,children:_jsx(_components.p,{children:`The example’s 1-based position in the run, fixing the order results are listed in.`})}),`\n`,_jsx(ResponseField,{name:`inputs`,type:`array`,required:true,id:`param-inputs-1`,children:_jsxs(_components.p,{children:[`The data handed to the agent, each ref carrying the CRC32C of the bytes scored. See `,_jsx(_components.a,{href:`#resolved-input-ref-examples-inputs`,children:`Resolved\ninput ref`}),` for the keys the service writes into each ref’s\n`,_jsx(_components.code,{children:`metadata`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`metadata`,type:`object`,id:`param-metadata-2`,children:_jsx(_components.p,{children:`The free-form tags the example was created with, if any.`})}),`\n`,_jsxs(Heading,{level:`3`,id:`resolved-input-ref-examples-inputs`,children:[`Resolved input ref (`,_jsx(_components.code,{children:`examples[].inputs[]`}),`)`]}),`\n`,_jsxs(_components.p,{children:[`Each input of a returned example is a data ref, with the same `,_jsx(_components.code,{children:`type`}),`, `,_jsx(_components.code,{children:`id`}),`, and `,_jsx(_components.code,{children:`format`}),` as in the\nrequest, plus the `,_jsx(_components.code,{children:`crc32c`}),` of the bytes scored. Its `,_jsx(_components.code,{children:`metadata`}),` carries the following keys, written\nby the service when the eval is created, never by the caller. Any other key is carried through\nuntouched.`]}),`\n`,_jsx(ResponseField,{name:`metadata.targets`,type:`object`,id:`param-metadata-targets`,children:_jsxs(_components.p,{children:[`Where this input’s ground truth is for every target the eval scores, with nothing left to\nconvention: a map of target name to that target’s `,_jsx(_components.a,{href:`#resolved-ground-truth-examples-inputs-metadata-targets-%3Ctarget-name%3E`,children:`resolved ground\ntruth`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`metadata.example_name`,type:`string`,id:`param-metadata-example-name`,children:_jsxs(_components.p,{children:[`The name of the example the input belongs to. `,_jsx(Badge,{color:`blue`,children:_jsx(_components.a,{href:`/release-notes/1.1.x#v1-1-13`,children:`v1.1.13+`})})]})}),`\n`,_jsxs(Heading,{level:`3`,id:`resolved-ground-truth-examples-inputs-metadata-targets-\u003ctarget-name>`,children:[`Resolved ground truth (`,_jsx(_components.code,{children:`examples[].inputs[].metadata.targets.\u003ctarget name>`}),`)`]}),`\n`,_jsxs(_components.p,{children:[`A ground truth with nothing left to convention: where its values are, and how they reduce. This is\nwhat an example’s `,_jsx(_components.a,{href:`#ground-truth-declaration-examples-ground_truth-%3Cname%3E`,children:`ground-truth declaration`}),`\nbecomes once laid over the defaults the run settled on.`]}),`\n`,_jsx(ResponseField,{name:`from`,type:`object`,required:true,id:`param-from-2`,children:_jsxs(_components.p,{children:[`Where the values are. Same forms as a ground-truth declaration’s `,_jsx(_components.code,{children:`from`}),`.`]})}),`\n`,_jsx(ResponseField,{name:`downsampling`,type:`string`,id:`param-downsampling-2`,children:_jsxs(_components.p,{children:[`For a target paired `,_jsx(_components.code,{children:`by_time`}),`, how the values of the records making up one analysis window\nreduce to the single value that window is scored against: the target’s own rule, unless the run\nor the example overrides it. Absent for a target paired any other way, which has nothing to\nreduce.`]})}),`\n`,_jsxs(RequestExample,{children:[_jsx(CodeBlock,{filename:`cURL - Conventional Examples`,numberOfLines:`11`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`11`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`curl`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -X`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` POST`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`$ATAI_API_ENDPOINT`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`/agents/evals\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -H`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Authorization: Bearer `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`$ATAI_API_KEY`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -H`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Content-Type: application/json\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -d`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` '{`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"name\": \"Pump A regression\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"blueprint_id\": \"blp_01jc9n7k3xf8mbq2v5t0ary6de\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"examples\": [`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` {\"inputs\": [{\"type\": \"file\", \"id\": \"file_abc123\", \"format\": \"csv\"}]},`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` {\"inputs\": [{\"type\": \"file\", \"id\": \"file_def456\", \"format\": \"csv\"}]}`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }'`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`cURL - With Overrides`,numberOfLines:`34`,language:`shellscript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`shellscript`,children:_jsxs(_components.code,{language:`shellscript`,numberOfLines:`34`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#DCDCAA`},children:`curl`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -X`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` POST`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`$ATAI_API_ENDPOINT`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`/agents/evals\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -H`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Authorization: Bearer `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`$ATAI_API_KEY`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -H`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Content-Type: application/json\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D7BA7D`},children:` \\\\`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:` -d`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` '{`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"name\": \"Pump A regression\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"blueprint_id\": \"blp_01jc9n7k3xf8mbq2v5t0ary6de\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"primary\": \"state.macro_f1\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"emit_predictions\": false,`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"targets\": {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"state\": {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"objectives\": [\"macro_f1\", {\"name\": \"recall_idle\", \"recall\": {\"class\": \"idle\"}}],`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"from\": {\"column\": \"ground_truth_state\"},`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"downsampling\": \"majority_vote\"`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"constraints\": [`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` {\"objective\": \"state.recall_idle\", \"min\": 0.6, \"max\": 0.95}`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ],`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"examples\": [`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"name\": \"site-a-morning\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"inputs\": [{\"type\": \"file\", \"id\": \"file_abc123\", \"format\": \"csv\"}],`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"metadata\": {\"site\": \"a\", \"shift\": \"morning\"}`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"name\": \"site-b-idle-run\",`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"inputs\": [{\"type\": \"file\", \"id\": \"file_def456\", \"format\": \"csv\"}],`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"ground_truth\": {`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"state\": {\"from\": {\"constant\": \"idle\"}}`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"metadata\": {\"site\": \"b\", \"shift\": \"night\"}`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` ]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` }'`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`Python`,numberOfLines:`27`,language:`python`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`python`,children:_jsxs(_components.code,{language:`python`,numberOfLines:`27`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` os`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`import`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` requests`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`base_url `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` os.environ[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"ATAI_API_ENDPOINT\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`api_key `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` os.environ[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"ATAI_API_KEY\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`response `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` requests.post(`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:` f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`base_url`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`/agents/evals\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` headers`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Authorization\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Bearer `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`api_key`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"Content-Type\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"application/json\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#953800`,\"--shiki-dark\":`#9CDCFE`},children:` json`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"name\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Pump A regression\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"blueprint_id\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"blp_01jc9n7k3xf8mbq2v5t0ary6de\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` \"examples\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"inputs\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [{`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"type\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"file\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"id\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"file_abc123\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"format\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"csv\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}]},`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"inputs\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [{`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"type\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"file\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"id\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"file_def456\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"format\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"csv\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}]},`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ],`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` response.status_code `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`==`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 201`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` evaluation `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:`=`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` response.json()`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Created `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`evaluation[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'id'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` (`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`evaluation[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'status'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`) scoring `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`evaluation[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'primary'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`else`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`:`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#DCDCAA`},children:` print`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`f`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Error: `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`{`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`response.json()[`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'errors'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`)`})]}),`\n`]})})}),_jsx(CodeBlock,{filename:`JavaScript`,numberOfLines:`23`,language:`javascript`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`javascript`,children:_jsxs(_components.code,{language:`javascript`,numberOfLines:`23`,children:[_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`const`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4FC1FF`},children:` response`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` await`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:` fetch`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:\"`\"}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:\"${\"}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`process`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`env`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4FC1FF`},children:`ATAI_API_ENDPOINT`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:\"/agents/evals`\"}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` method:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'POST'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` headers:`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'Authorization'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:\" `Bearer \"}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:\"${\"}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`process`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`env`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4FC1FF`},children:`ATAI_API_KEY`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:\"`\"}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'Content-Type'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'application/json'`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` body:`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4FC1FF`},children:` JSON`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:`stringify`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`({`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` name:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'Pump A regression'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` blueprint_id:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'blp_01jc9n7k3xf8mbq2v5t0ary6de'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` examples:`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` { `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`inputs:`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [{ `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`type:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'file'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`id:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'file_abc123'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`format:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'csv'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }] },`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` { `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`inputs:`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` [{ `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`type:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'file'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`id:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'file_def456'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`format:`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` 'csv'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }] }`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` })`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`});`})}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#569CD6`},children:`const`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#4FC1FF`},children:` body`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` =`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:` await`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` response`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:`json`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`();`})]}),`\n`,_jsx(_components.span,{className:`line`}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`if`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` (`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`response`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`status`}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#D4D4D4`},children:` ===`}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:` 201`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`) {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` console`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:`log`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:\"`Created \"}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:\"${\"}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`body`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`id`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:` (`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:\"${\"}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`body`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`status`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`) scoring `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:\"${\"}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`body`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`primary`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#569CD6`},children:`}`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:\"`\"}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`);`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`} `}),_jsx(_components.span,{style:{color:`#CF222E`,\"--shiki-dark\":`#C586C0`},children:`else`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:` console`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#8250DF`,\"--shiki-dark\":`#DCDCAA`},children:`error`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`(`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`'Error:'`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`body`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`.`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#9CDCFE`},children:`errors`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`);`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})})]}),`\n`,_jsxs(ResponseExample,{children:[_jsx(CodeBlock,{filename:`201 - Created`,numberOfLines:`35`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`35`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"id\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"evl_01jcb0h2m6t4xr9nv3k7pdzs5y\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"name\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Pump A regression\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"org_id\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"org_01jc8m5r2vq9xt4bn7h3kdzs6w\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"blueprint_id\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"blp_01jc9n7k3xf8mbq2v5t0ary6de\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"primary\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"state.macro_f1\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"resolved_targets\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"state\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [`}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"macro_f1\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`, `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"accuracy\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`]`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"constraints\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [],`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"emit_predictions\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`true`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"examples\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"name\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"pump-a-week-1\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"ordinal\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#B5CEA8`},children:`1`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"inputs\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"type\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"file\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"id\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"file_abc123\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"format\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"csv\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"crc32c\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"AAAAAA==\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"metadata\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"targets\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: {`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"state\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: {`}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"from\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: {`}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"column\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"state\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}, `}),_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:`\"downsampling\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"last_record\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` },`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"example_name\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"file_abc123\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ],`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"status\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"pending\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"created_by\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"usr_01jc8m4p3rt6vx9qn2h5kdzb7y\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"created_at\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"2026-09-18T11:24:03Z\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`400 - Invalid request`,numberOfLines:`10`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`10`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"errors\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"code\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"\u003cerror_code>\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"message\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Invalid request.\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"suggestion\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`null`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"error_uid\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"err-xxxxxxxx\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`403 - Blueprint not available`,numberOfLines:`10`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`10`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"errors\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"code\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"capability_locked\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"message\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Blueprint not available to the caller's organization.\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"suggestion\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`null`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"error_uid\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"err-xxxxxxxx\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})}),_jsx(CodeBlock,{filename:`404 - Blueprint not found`,numberOfLines:`10`,language:`json`,children:_jsx(_components.pre,{className:`shiki shiki-themes github-light-default dark-plus`,style:{backgroundColor:`#ffffff`,\"--shiki-dark-bg\":`#0B0C0E`,color:`#1f2328`,\"--shiki-dark\":`#D4D4D4`},language:`json`,children:_jsxs(_components.code,{language:`json`,numberOfLines:`10`,children:[_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`{`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"errors\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: [`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` {`})}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"code\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"\u003cerror_code>\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"message\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"Blueprint not found.\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"suggestion\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0550AE`,\"--shiki-dark\":`#569CD6`},children:`null`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`,`})]}),`\n`,_jsxs(_components.span,{className:`line`,children:[_jsx(_components.span,{style:{color:`#116329`,\"--shiki-dark\":`#9CDCFE`},children:` \"error_uid\"`}),_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`: `}),_jsx(_components.span,{style:{color:`#0A3069`,\"--shiki-dark\":`#CE9178`},children:`\"err-xxxxxxxx\"`})]}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` }`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:` ]`})}),`\n`,_jsx(_components.span,{className:`line`,children:_jsx(_components.span,{style:{color:`#1F2328`,\"--shiki-dark\":`#D4D4D4`},children:`}`})}),`\n`]})})})]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"codeStyling":"system","config":{},"pageMetadata":{"title":"Create Eval","api":"POST /agents/evals","description":"Score a blueprint against a set of labeled examples","tag":"v1.1.12+","href":"/api-reference/agents/evals/create-eval"}}}},"mdxExtracts":{"tableOfContents":[{"title":"Overview","slug":"overview","depth":2,"children":[]},{"title":"Request","slug":"request","depth":2,"children":[{"title":"Example object (examples[])","slug":"example-object-examples","depth":3,"children":[]},{"title":"Data ref (examples[].inputs[])","slug":"data-ref-examples-inputs","depth":3,"children":[]},{"title":"Ground-truth declaration (examples[].ground_truth.\u003cname>)","slug":"ground-truth-declaration-examples-ground_truth-\u003cname>","depth":3,"children":[]},{"title":"Target override (targets.\u003ctarget name>)","slug":"target-override-targets-\u003ctarget-name>","depth":3,"children":[]},{"title":"Constraint (constraints[])","slug":"constraint-constraints","depth":3,"children":[]}]},{"title":"Response","slug":"response","depth":2,"children":[{"title":"Resolved example object (examples[])","slug":"resolved-example-object-examples","depth":3,"children":[]},{"title":"Resolved input ref (examples[].inputs[])","slug":"resolved-input-ref-examples-inputs","depth":3,"children":[]},{"title":"Resolved ground truth (examples[].inputs[].metadata.targets.\u003ctarget name>)","slug":"resolved-ground-truth-examples-inputs-metadata-targets-\u003ctarget-name>","depth":3,"children":[]}]}],"codeExamples":{"request":{"type":"RequestExample","children":[{"filename":"cURL - Conventional Examples","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"shellscript\">\u003ccode language=\"shellscript\" numberOfLines=\"11\">\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#DCDCAA\">curl\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -X\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> POST\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">$ATAI_API_ENDPOINT\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">/agents/evals\"\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D7BA7D\"> \\\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -H\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"Authorization: Bearer \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">$ATAI_API_KEY\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D7BA7D\"> \\\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -H\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"Content-Type: application/json\"\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D7BA7D\"> \\\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -d\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> '{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"name\": \"Pump A regression\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"blueprint_id\": \"blp_01jc9n7k3xf8mbq2v5t0ary6de\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"examples\": [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> {\"inputs\": [{\"type\": \"file\", \"id\": \"file_abc123\", \"format\": \"csv\"}]},\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> {\"inputs\": [{\"type\": \"file\", \"id\": \"file_def456\", \"format\": \"csv\"}]}\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> ]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> }'\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"cURL - With Overrides","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"shellscript\">\u003ccode language=\"shellscript\" numberOfLines=\"34\">\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#DCDCAA\">curl\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -X\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> POST\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">$ATAI_API_ENDPOINT\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">/agents/evals\"\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D7BA7D\"> \\\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -H\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"Authorization: Bearer \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">$ATAI_API_KEY\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D7BA7D\"> \\\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -H\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"Content-Type: application/json\"\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D7BA7D\"> \\\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\"> -d\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> '{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"name\": \"Pump A regression\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"blueprint_id\": \"blp_01jc9n7k3xf8mbq2v5t0ary6de\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"primary\": \"state.macro_f1\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"emit_predictions\": false,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"targets\": {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"state\": {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"objectives\": [\"macro_f1\", {\"name\": \"recall_idle\", \"recall\": {\"class\": \"idle\"}}],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"from\": {\"column\": \"ground_truth_state\"},\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"downsampling\": \"majority_vote\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"constraints\": [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> {\"objective\": \"state.recall_idle\", \"min\": 0.6, \"max\": 0.95}\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> ],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"examples\": [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"name\": \"site-a-morning\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"inputs\": [{\"type\": \"file\", \"id\": \"file_abc123\", \"format\": \"csv\"}],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"metadata\": {\"site\": \"a\", \"shift\": \"morning\"}\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"name\": \"site-b-idle-run\",\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"inputs\": [{\"type\": \"file\", \"id\": \"file_def456\", \"format\": \"csv\"}],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"ground_truth\": {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"state\": {\"from\": {\"constant\": \"idle\"}}\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"metadata\": {\"site\": \"b\", \"shift\": \"night\"}\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> ]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> }'\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"Python","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"python\">\u003ccode language=\"python\" numberOfLines=\"27\">\u003cspan class=\"line\">\u003cspan style=\"color:#CF222E;--shiki-dark:#C586C0\">import\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> os\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#CF222E;--shiki-dark:#C586C0\">import\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> requests\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">base_url \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> os.environ[\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"ATAI_API_ENDPOINT\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">api_key \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> os.environ[\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"ATAI_API_KEY\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">response \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> requests.post(\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\"> f\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">{\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">base_url\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">/agents/evals\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#9CDCFE\"> headers\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"Authorization\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">f\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Bearer \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">{\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">api_key\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"Content-Type\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"application/json\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#953800;--shiki-dark:#9CDCFE\"> json\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"name\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Pump A regression\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"blueprint_id\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"blp_01jc9n7k3xf8mbq2v5t0ary6de\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> \"examples\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> {\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"inputs\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [{\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"type\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"file\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"id\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"file_abc123\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"format\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"csv\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}]},\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> {\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"inputs\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [{\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"type\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"file\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"id\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"file_def456\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"format\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"csv\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}]},\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> ],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">)\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#CF222E;--shiki-dark:#C586C0\">if\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> response.status_code \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">==\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#B5CEA8\"> 201\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">:\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> evaluation \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\">=\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> response.json()\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#DCDCAA\"> print\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">(\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">f\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Created \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">{\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">evaluation[\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">'id'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">]\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> (\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">{\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">evaluation[\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">'status'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">]\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">) scoring \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">{\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">evaluation[\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">'primary'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">]\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">)\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#CF222E;--shiki-dark:#C586C0\">else\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">:\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0550AE;--shiki-dark:#DCDCAA\"> print\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">(\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">f\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Error: \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">{\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">response.json()[\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">'errors'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">]\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">)\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"JavaScript","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"javascript\">\u003ccode language=\"javascript\" numberOfLines=\"23\">\u003cspan class=\"line\">\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">const\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#4FC1FF\"> response\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\"> =\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#C586C0\"> await\u003c/span>\u003cspan style=\"color:#8250DF;--shiki-dark:#DCDCAA\"> fetch\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">(\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">`\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">${\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">process\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">env\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#4FC1FF\">ATAI_API_ENDPOINT\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">/agents/evals`\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\"> method:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'POST'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\"> headers:\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'Authorization'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> `Bearer \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">${\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">process\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">env\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#4FC1FF\">ATAI_API_KEY\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">`\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'Content-Type'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'application/json'\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\"> body:\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#4FC1FF\"> JSON\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#8250DF;--shiki-dark:#DCDCAA\">stringify\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">({\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\"> name:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'Pump A regression'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\"> blueprint_id:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'blp_01jc9n7k3xf8mbq2v5t0ary6de'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\"> examples:\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> { \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">inputs:\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> [{ \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">type:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'file'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">id:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'file_abc123'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">format:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'csv'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> }] },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> { \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">inputs:\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> [{ \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">type:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'file'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">id:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'file_def456'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">format:\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> 'csv'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> }] }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> ]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> })\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">});\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#CF222E;--shiki-dark:#569CD6\">const\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#4FC1FF\"> body\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\"> =\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#C586C0\"> await\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\"> response\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#8250DF;--shiki-dark:#DCDCAA\">json\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">();\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#CF222E;--shiki-dark:#C586C0\">if\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> (\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">response\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">status\u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#D4D4D4\"> ===\u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#B5CEA8\"> 201\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">) {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\"> console\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#8250DF;--shiki-dark:#DCDCAA\">log\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">(\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">`Created \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">${\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">body\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">id\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\"> (\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">${\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">body\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">status\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">) scoring \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">${\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">body\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">primary\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#569CD6\">}\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">`\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">);\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">} \u003c/span>\u003cspan style=\"color:#CF222E;--shiki-dark:#C586C0\">else\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\"> console\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#8250DF;--shiki-dark:#DCDCAA\">error\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">(\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">'Error:'\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">body\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">.\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#9CDCFE\">errors\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">);\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"}],"attributes":[]},"response":{"type":"ResponseExample","children":[{"filename":"201 - Created","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"json\">\u003ccode language=\"json\" numberOfLines=\"35\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"id\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"evl_01jcb0h2m6t4xr9nv3k7pdzs5y\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"name\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Pump A regression\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"org_id\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"org_01jc8m5r2vq9xt4bn7h3kdzs6w\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"blueprint_id\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"blp_01jc9n7k3xf8mbq2v5t0ary6de\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"primary\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"state.macro_f1\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"resolved_targets\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"state\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [\u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"macro_f1\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">, \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"accuracy\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"constraints\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"emit_predictions\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">true\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"examples\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"name\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"pump-a-week-1\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"ordinal\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#B5CEA8\">1\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"inputs\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"type\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"file\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"id\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"file_abc123\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"format\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"csv\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"crc32c\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"AAAAAA==\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"metadata\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"targets\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"state\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: {\u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"from\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: {\u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"column\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"state\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}, \u003c/span>\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\">\"downsampling\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"last_record\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> },\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"example_name\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"file_abc123\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> ]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> ],\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"status\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"pending\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"created_by\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"usr_01jc8m4p3rt6vx9qn2h5kdzb7y\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"created_at\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"2026-09-18T11:24:03Z\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"400 - Invalid request","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"json\">\u003ccode language=\"json\" numberOfLines=\"10\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"errors\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"code\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"\u0026#x3C;error_code>\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"message\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Invalid request.\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"suggestion\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">null\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"error_uid\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"err-xxxxxxxx\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> ]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"403 - Blueprint not available","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"json\">\u003ccode language=\"json\" numberOfLines=\"10\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"errors\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"code\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"capability_locked\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"message\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Blueprint not available to the caller's organization.\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"suggestion\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">null\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"error_uid\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"err-xxxxxxxx\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> ]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"},{"filename":"404 - Blueprint not found","html":"\u003cpre class=\"shiki shiki-themes github-light-default dark-plus\" style=\"background-color:#ffffff;--shiki-dark-bg:#0B0C0E;color:#1f2328;--shiki-dark:#D4D4D4\" language=\"json\">\u003ccode language=\"json\" numberOfLines=\"10\">\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">{\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"errors\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: [\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> {\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"code\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"\u0026#x3C;error_code>\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"message\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"Blueprint not found.\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"suggestion\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0550AE;--shiki-dark:#569CD6\">null\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">,\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#116329;--shiki-dark:#9CDCFE\"> \"error_uid\"\u003c/span>\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">: \u003c/span>\u003cspan style=\"color:#0A3069;--shiki-dark:#CE9178\">\"err-xxxxxxxx\"\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> }\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\"> ]\u003c/span>\u003c/span>\n\u003cspan class=\"line\">\u003cspan style=\"color:#1F2328;--shiki-dark:#D4D4D4\">}\u003c/span>\u003c/span>\n\u003c/code>\u003c/pre>"}],"attributes":[]}},"endpoint":{"title":"Create Eval","path":"/agents/evals","servers":[{"url":"https://platform.u1.archetypeai.app/api"}],"method":"post","request":{"security":[{"title":"Bearer Auth","parameters":{"header":{"Authorization":{"type":"http","scheme":"bearer","description":"Bearer authentication header of the form `Bearer \u003ctoken>`, where `\u003ctoken>` is your auth token."}},"query":{},"cookie":{}}}],"parameters":{"query":{},"header":{},"cookie":{},"path":{}},"body":{"application/json":{"schemaArray":[{"type":"object","properties":{"blueprint_id":[{"type":"string","description":"The blueprint to evaluate — a `blp_` id. Must exist and be visible to the caller’s organization.","required":true}],"examples":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"The examples to score. Must be non-empty. See Example object .","required":true}],"name":[{"type":"string","description":"The name by which this example is identified in the run’s results. Defaults to the file name of its input, without the extension. Names must be unique within a request: results are keyed by name, and two examples sharing one cannot be told apart in a report."}],"primary":[{"type":"string","description":"The run’s headline, as `\u003ctarget>.\u003cobjective>` — e.g. `state.macro_f1` . Defaults to the blueprint’s own. Must name a target this run scores and an objective it computes for it."}],"emit_predictions":[{"type":"boolean","description":"Emit the predictions NDJSON artifacts alongside the metrics report: one per target, with a row per scored item. The artifacts are what make a score explainable. Set `false` to skip them when the input is large enough that they are not worth their size."}],"constraints":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"Bounds the run reports a verdict on, replacing those the blueprint declares. Omit it to use the blueprint’s; send `[]` for none. Each entry names an objective the run computes, with a `min` , a `max` , or both. See Constraint . Missing a constraint does not fail the eval: Get Eval reports each constraint’s verdict and whether they all hold."}],"targets":[{"type":"object","properties":{},"description":"Per-target overrides for this run, keyed by target name. Every field of an entry is optional and falls back to what the blueprint’s target declares, so a run that wants nothing different omits this entirely — and one that does need a change states it once here rather than on every file. See Target override . A key must name a target the blueprint declares. An entry’s `objectives` replaces the target’s own list for this run, and may include objectives the target does not declare."}],"inputs":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"The data handed to the agent for this example, in order: `file` references, each file used at most once in a request. Each entry is a data ref .","required":true}],"ground_truth":[{"type":"object","properties":{},"description":"The annotation supplied with this example, keyed by a name of your choosing: where its values are and, where they are not recorded at the cadence the agent emits at, how they reduce. Each value is a ground-truth declaration . This says nothing about which target it serves; that’s done by `binding` , so the same annotation can be written once and scored against different agents. Omit `ground_truth` entirely to use the convention: each target’s labels are a column of the input named after the target."}],"binding":[{"type":"object","properties":{},"description":"Which targets each annotation serves, keyed by the annotation’s name; each value is an array of target names. Omit it whenever the names line up: an annotation named after a target serves that target, and a single annotation serves a single target whatever either is called."}],"metadata":[{"type":"object","properties":{},"description":"Free-form detail about this ref. Any key is carried through untouched."}],"type":[{"type":"string","description":"Storage kind, e.g. `file` or `dataset` .","required":true}],"id":[{"type":"string","description":"Identifier within that storage kind, e.g. `file_abc123` .","required":true}],"format":[{"type":"string","description":"Optional format hint, e.g. `csv` ."}],"crc32c":[{"type":"string","description":"Whole-file CRC32C checksum of the referenced bytes, base64-encoded exactly as S3 emits it. You don’t need to send it: when the eval is created, it’s pinned from the checksum the data service stores for the file, so the eval records exactly which bytes it scored even if a filename is later reused. `null` when the data service has no checksum for the file."}],"from":[{"type":"object","properties":{},"description":"Where every input of this run keeps the target’s labels, when they all agree and disagree with the target’s name. An input may still override it. Same forms as a ground-truth declaration’s `from` ."}],"downsampling":[{"type":"string","description":"For a target paired `by_time` only: how the values of one analysis window reduce to the single value it is scored against, for every example of this run: `last_record` or `majority_vote` . An example may still override it."}],"objectives":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"The objectives to compute for the target in this run, replacing those the target declares. The list may include objectives the target does not declare. Each entry is one of: the name of an objective the target declares, which keeps that declaration a catalog objective that takes no argument, named bare: `macro_f1` , `accuracy` , `balanced_accuracy` , `foreground_macro_f1` , `detection_recall` , or `detection_precision` an objective written in full, as an object with the catalog objective as its key and its arguments as the value, plus an optional `name` The per-class objectives `precision` , `recall` , and `f1` take a required `class` argument, one of the target’s labels, so they are always written in full: for example, `{\"name\": \"recall_cold\", \"recall\": {\"class\": \"cold\"}}` . `name` is what `primary` and the metrics report refer to the objective by, and defaults to the catalog objective’s name. Names must be unique within a target, so give each one a `name` to compute the same per-class objective for several classes. Written in full, an objective may reuse a name the target declares, and then replaces that declaration for the run. `macro_f1` is shorthand for `{\"macro_f1\": {}}` . Written in full, an objective may also set `if_undefined` : the number it takes wherever it has no value of its own, such as an example with nothing to compute it from. For example, `{\"mean_iou\": {}, \"if_undefined\": 1}` . The value must be within the range of values the objective takes, or the eval is rejected when it is created. The metrics report lists the objectives that took this value in `defaulted` . The catalog is closed: an objective outside it is rejected when the eval is created, rather than producing an unscored run."}],"objective":[{"type":"string","description":"The objective bounded, as `\u003ctarget>.\u003cobjective>` , naming an objective the run computes for that target, with `targets` applied. For example, `state.macro_f1` , or `state.recall_idle` for a per-class objective added through `targets` .","required":true}],"min":[{"type":"number","description":"The lowest acceptable value, inclusive."}],"max":[{"type":"number","description":"The highest acceptable value, inclusive."}]}}],"examples":{"example":{"value":{"blueprint_id":"\u003cstring>","examples":[{}],"name":"\u003cstring>","primary":"\u003cstring>","emit_predictions":true,"constraints":[{}],"targets":{},"inputs":[{}],"ground_truth":{},"binding":{},"metadata":{},"type":"\u003cstring>","id":"\u003cstring>","format":"\u003cstring>","crc32c":"\u003cstring>","from":{},"downsampling":"\u003cstring>","objectives":[{}],"objective":"\u003cstring>","min":123,"max":123}}}}}},"response":{"200":{"application/json":{"schemaArray":[{"type":"object","properties":{"id":[{"type":"string","description":"The eval’s `evl_` id.","required":true}],"name":[{"type":"string","description":"How this example is identified in the run’s results.","required":true}],"org_id":[{"type":"string","description":"The organization that owns the eval.","required":true}],"blueprint_id":[{"type":"string","description":"The blueprint being evaluated.","required":true}],"primary":[{"type":"string","description":"The run’s headline, as `\u003ctarget>.\u003cobjective>` — resolved at creation, so this is what was actually scored rather than what was asked for.","required":true}],"resolved_targets":[{"type":"object","properties":{},"description":"The objectives this run computes, keyed by target name — the blueprint’s own declaration folded together with the request’s per-target overrides, resolved at creation. Each objective is in the same form as in a target override : a bare name, or written in full with its arguments. `primary` names one of these, as `\u003ctarget>.\u003cname>` . Read this rather than the blueprint’s `metrics` to know what was scored, since a request may replace a target’s objectives.","required":true}],"constraints":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"The bounds the run reports a verdict on, resolved at creation: the request’s, or else the blueprint’s. Empty when there are none.","required":true}],"emit_predictions":[{"type":"boolean","description":"Whether the run emits the predictions artifacts alongside the metrics report, one per target, as the request asked.","required":true}],"examples":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"The examples the eval was created against, as resolved: each carries the name its results are keyed by, and inputs stamped with the CRC32C of the bytes scored. See Resolved example object .","required":true}],"status":[{"type":"string","description":"The eval’s current lifecycle status; `pending` on creation.","required":true}],"project_id":[{"type":"string","description":"The project the eval is in: its blueprint’s. This value is `null` outside any project."}],"created_by":[{"type":"string","description":"Subject id ( `usr_...` or `key_...` ) that created this eval.","required":true}],"created_at":[{"type":"string","description":"When the eval was created (date-time).","required":true}],"started_at":[{"type":"string","description":"When scoring began. Absent while `pending` ."}],"completed_at":[{"type":"string","description":"When the eval reached a terminal status. Absent until then."}],"metrics_report":[{"type":"object","properties":{},"description":"What the run measured. Present once the eval is `completed` . See Get Eval for its shape."}],"output_artifacts":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"Refs to the files the run produced, filled in while the eval runs. Each `id` is a data-service file id, so an artifact is downloadable through the files API."}],"error":[{"type":"string","description":"Failure detail; `null` unless the eval failed."}],"ordinal":[{"type":"integer","description":"The example’s 1-based position in the run, fixing the order results are listed in.","required":true}],"inputs":[{"type":"array","items":[{"type":"object","properties":{}}],"description":"The data handed to the agent, each ref carrying the CRC32C of the bytes scored. See Resolved input ref for the keys the service writes into each ref’s `metadata` .","required":true}],"metadata":[{"type":"object","properties":{},"description":"The free-form tags the example was created with, if any."}],"metadata.targets":[{"type":"object","properties":{},"description":"Where this input’s ground truth is for every target the eval scores, with nothing left to convention: a map of target name to that target’s resolved ground truth ."}],"metadata.example_name":[{"type":"string","description":"The name of the example the input belongs to."}],"from":[{"type":"object","properties":{},"description":"Where the values are. Same forms as a ground-truth declaration’s `from` .","required":true}],"downsampling":[{"type":"string","description":"For a target paired `by_time` , how the values of the records making up one analysis window reduce to the single value that window is scored against: the target’s own rule, unless the run or the example overrides it. Absent for a target paired any other way, which has nothing to reduce."}]}}],"examples":{"example":{"value":{"id":"\u003cstring>","name":"\u003cstring>","org_id":"\u003cstring>","blueprint_id":"\u003cstring>","primary":"\u003cstring>","resolved_targets":{},"constraints":[{}],"emit_predictions":true,"examples":[{}],"status":"\u003cstring>","project_id":"\u003cstring>","created_by":"\u003cstring>","created_at":"\u003cstring>","started_at":"\u003cstring>","completed_at":"\u003cstring>","metrics_report":{},"output_artifacts":[{}],"error":"\u003cstring>","ordinal":123,"inputs":[{}],"metadata":{},"metadata.targets":{},"metadata.example_name":"\u003cstring>","from":{},"downsampling":"\u003cstring>"}}}}}},"deprecated":false}},"jsLinks":[],"subdomain":"archetypeai-bd6fb3cf","siteName":"Archetype AI Documentation","title":"Create Eval - Archetype AI Documentation","relatedPagesConfig":{"enabled":true,"limit":5,"mode":"manual"},"navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-11-12T16:08:42.538Z","originalFileLocation":"api-reference/agents/evals/create-eval.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=null;if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"system\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"description\" content=\"Score a blueprint against a set of labeled examples\">\u003cmeta data-mint-head name=\"application-name\" content=\"Archetype AI Documentation\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Create Eval - Archetype AI Documentation\">\u003cmeta data-mint-head property=\"og:description\" content=\"Score a blueprint against a set of labeled examples\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Archetype AI Documentation\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://archetypeai-bd6fb3cf.mintlify.site/_mintlify/api/og/v2?division=Evals\u0026amp;title=Create+Eval\u0026amp;description=Score+a+blueprint+against+a+set+of+labeled+examples\u0026amp;theme=baca9d5089929997d797e9d7\u0026amp;signature=579bfc2b5973a276f4d1e98eb3905e21e716c925cac2d47c529acc1854f9ab95\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Create Eval - Archetype AI Documentation\">\u003cmeta data-mint-head name=\"twitter:description\" content=\"Score a blueprint against a set of labeled examples\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://archetypeai-bd6fb3cf.mintlify.site/_mintlify/api/og/v2?division=Evals\u0026amp;title=Create+Eval\u0026amp;description=Score+a+blueprint+against+a+set+of+labeled+examples\u0026amp;theme=baca9d5089929997d797e9d7\u0026amp;signature=579bfc2b5973a276f4d1e98eb3905e21e716c925cac2d47c529acc1854f9ab95\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/archetypeai-bd6fb3cf/k6xQEXnXaSGa-p6g/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Archetype AI Documentation\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#0047FF\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/api-reference/agents/evals/create-eval.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.archetypeai.app/#organization\",\"name\":\"Archetype AI Documentation\",\"url\":\"https://docs.archetypeai.app\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/archetypeai-bd6fb3cf/Fml2i7FmPyxjLCP7/logo/light.svg?fit=max\u0026auto=format\u0026n=Fml2i7FmPyxjLCP7\u0026q=85\u0026s=4b1f94a0bc6c65c3905d1a55d725b05f\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.archetypeai.app#website\",\"name\":\"Archetype AI Documentation\",\"url\":\"https://docs.archetypeai.app\",\"publisher\":{\"@id\":\"https://docs.archetypeai.app/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval#webpage\",\"url\":\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval\",\"name\":\"Create Eval\",\"description\":\"Score a blueprint against a set of labeled examples\",\"dateModified\":\"2026-11-12T16:08:42.538Z\",\"isPartOf\":{\"@id\":\"https://docs.archetypeai.app#website\"},\"breadcrumb\":{\"@id\":\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"API Reference\",\"item\":\"https://docs.archetypeai.app/api-reference/introduction\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Agents API\",\"item\":\"https://docs.archetypeai.app/api-reference/agents/about\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Evals\",\"item\":\"https://docs.archetypeai.app/api-reference/agents/evals/cancel-eval\"},{\"@type\":\"ListItem\",\"position\":4,\"name\":\"Create Eval\",\"item\":\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval\"}]},{\"@type\":[\"TechArticle\",\"APIReference\"],\"@id\":\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval#apireference\",\"headline\":\"Create Eval\",\"name\":\"Create Eval\",\"description\":\"Score a blueprint against a set of labeled examples\",\"url\":\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval\",\"mainEntityOfPage\":{\"@id\":\"https://docs.archetypeai.app/api-reference/agents/evals/create-eval#webpage\"},\"dateModified\":\"2026-11-12T16:08:42.538Z\",\"publisher\":{\"@id\":\"https://docs.archetypeai.app/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.archetypeai.app#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Archetype AI Documentation\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Archetype AI Documentation\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/api-reference/agents/evals/create-eval.md","siteName":"Archetype AI Documentation","nav":{"global":null,"tabs":[{"tab":"Documentation","pages":[{"title":"Documentation","href":"/introduction/overview"}]},{"tab":"API Reference","directory":"accordion","groups":[{"group":"API Reference","public":true,"pages":[{"title":"Overview","href":"/api-reference/introduction"},{"title":"Quick Reference","href":"/api-reference/quick-reference"},{"title":"Error Handling","href":"/api-reference/errors"},{"title":"Direct Query","api":"POST /lens/query","deprecated":true,"href":"/api-reference/query"},{"group":"Agents API","root":{"title":"About the Agents API","sidebarTitle":"About","href":"/api-reference/agents/about"},"pages":[{"group":"Blueprints","pages":[{"title":"Copy Blueprint","api":"POST /agents/blueprints/{reference}/copy","tag":"v2.0.0+","href":"/api-reference/agents/blueprints/copy-blueprint"},{"title":"Create Blueprint","api":"POST /agents/blueprints","tag":"v1.1.9+","href":"/api-reference/agents/blueprints/create-blueprint"},{"title":"Delete Blueprint","api":"DELETE /agents/blueprints/{reference}","tag":"v1.1.9+","href":"/api-reference/agents/blueprints/delete-blueprint"},{"title":"Get Blueprint","api":"GET /agents/blueprints/{reference}","tag":"v1.1.9+","href":"/api-reference/agents/blueprints/get-blueprint"},{"title":"List Blueprint Versions","api":"GET /agents/blueprints/{reference}/versions","tag":"v1.1.12+","href":"/api-reference/agents/blueprints/list-blueprint-versions"},{"title":"List Blueprints","api":"GET /agents/blueprints","tag":"v1.1.9+","href":"/api-reference/agents/blueprints/list-blueprints"}]},{"group":"Bundles","pages":[{"title":"Create Bundle","api":"POST /agents/bundles","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/bundles/create-bundle"},{"title":"Delete Bundle","api":"DELETE /agents/bundles/{bundle_id}","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/bundles/delete-bundle"},{"title":"Get Bundle","api":"GET /agents/bundles/{bundle_id}","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/bundles/get-bundle"},{"title":"List Bundles","api":"GET /agents/bundles","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/bundles/list-bundles"},{"title":"Run Bundle","api":"POST /agents/bundles/{bundle_id}/run","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/bundles/run-bundle"}]},{"group":"Deployments","pages":[{"title":"Create Deployment","api":"POST /api/agents/deployments","tag":"v2.0.1+","href":"/api-reference/agents/deployments/create-deployment"},{"title":"Delete Deployment","api":"DELETE /api/agents/deployments/{deployment_id}","tag":"v2.0.1+","href":"/api-reference/agents/deployments/delete-deployment"},{"title":"Get Deployment","api":"GET /api/agents/deployments/{deployment_id}","tag":"v2.0.1+","href":"/api-reference/agents/deployments/get-deployment"},{"title":"List Deployments","api":"GET /api/agents/deployments","tag":"v2.0.1+","href":"/api-reference/agents/deployments/list-deployments"},{"title":"Stop Deployment","api":"POST /api/agents/deployments/{deployment_id}/stop","tag":"v2.0.1+","href":"/api-reference/agents/deployments/stop-deployment"},{"title":"Update Deployment","api":"PATCH /api/agents/deployments/{deployment_id}","tag":"v2.0.1+","href":"/api-reference/agents/deployments/update-deployment"}]},{"group":"Evals","pages":[{"title":"Cancel Eval","api":"POST /agents/evals/{eval_id}/cancel","description":"Cancel a pending or running eval","tag":"v1.1.12+","href":"/api-reference/agents/evals/cancel-eval"},{"title":"Create Eval","api":"POST /agents/evals","tag":"v1.1.12+","href":"/api-reference/agents/evals/create-eval"},{"title":"Delete Eval","api":"DELETE /agents/evals/{eval_id}","description":"Delete a terminal eval by id","tag":"v1.1.12+","href":"/api-reference/agents/evals/delete-eval"},{"title":"Get Eval","api":"GET /agents/evals/{eval_id}","tag":"v1.1.12+","href":"/api-reference/agents/evals/get-eval"},{"title":"List Eval Examples","api":"GET /agents/evals/{eval_id}/examples","tag":"v1.1.12+","href":"/api-reference/agents/evals/list-eval-examples"},{"title":"List Evals","api":"GET /agents/evals","tag":"v1.1.12+","href":"/api-reference/agents/evals/list-evals"}]},{"group":"Instances","pages":[{"title":"Cancel Agent","api":"POST /agents/instances/{agent_id}/cancel","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/instances/cancel-agent"},{"title":"Create Agent","api":"POST /agents/instances","tag":"v2.0.0+","deprecated":true,"href":"/api-reference/agents/instances/create-agent"},{"title":"Delete Agent","api":"DELETE /agents/instances/{agent_id}","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/instances/delete-agent"},{"title":"Get Agent Logs","api":"GET /agents/instances/{agent_id}/logs","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/instances/get-agent-logs"},{"title":"Get Agent Results","api":"GET /agents/instances/{agent_id}/results","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/instances/get-agent-results"},{"title":"Get Agent","api":"GET /agents/instances/{agent_id}","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/instances/get-agent"},{"title":"List Agent Events","api":"GET /agents/instances/{agent_id}/events","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/instances/list-agent-events"},{"title":"List Agents","api":"GET /agents/instances","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/instances/list-agents"},{"title":"Pause Agent","api":"POST /agents/instances/{agent_id}/pause","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/instances/pause-agent"},{"title":"Connect to Agent","api":"GET /agents/instances/{agent_id}/connect","tag":"v1.1.13+","deprecated":true,"href":"/api-reference/agents/instances/connect-agent"},{"title":"Query Agent","api":"POST /agents/instances/{agent_id}/query","tag":"v1.1.12+","deprecated":true,"href":"/api-reference/agents/instances/query-agent"},{"title":"Resume Agent","api":"POST /agents/instances/{agent_id}/resume","tag":"v1.1.9+","deprecated":true,"href":"/api-reference/agents/instances/resume-agent"}]},{"group":"Nodes","pages":[{"title":"List Node Registry","api":"GET /agents/nodes/registry","tag":"v1.1.9+","href":"/api-reference/agents/nodes/list-node-registry"}]},{"group":"Optimizations","pages":[{"title":"Cancel Optimization","api":"POST /agents/optimizations/{optimization_id}/cancel","tag":"v1.1.12+","href":"/api-reference/agents/optimizations/cancel-optimization"},{"title":"Create Optimization","api":"POST /agents/optimizations","tag":"v1.1.12+","href":"/api-reference/agents/optimizations/create-optimization"},{"title":"Delete Optimization","api":"DELETE /agents/optimizations/{optimization_id}","tag":"v1.1.12+","href":"/api-reference/agents/optimizations/delete-optimization"},{"title":"Get Optimization","api":"GET /agents/optimizations/{optimization_id}","tag":"v1.1.12+","href":"/api-reference/agents/optimizations/get-optimization"},{"title":"List Optimization Trials","api":"GET /agents/optimizations/{optimization_id}/trials","tag":"v1.1.12+","href":"/api-reference/agents/optimizations/list-optimization-trials"},{"title":"List Optimizations","api":"GET /agents/optimizations","tag":"v1.1.12+","href":"/api-reference/agents/optimizations/list-optimizations"},{"title":"Promote Trial","api":"POST /agents/optimizations/{optimization_id}/trials/{trial_id}/promote","tag":"v1.1.12+","href":"/api-reference/agents/optimizations/promote-trial"}]},{"group":"Packages","pages":[{"title":"Create Package","api":"POST /agents/packages","tag":"v2.0.0+","href":"/api-reference/agents/packages/create-package"},{"title":"Delete Package","api":"DELETE /agents/packages/{package_id}","tag":"v2.0.0+","href":"/api-reference/agents/packages/delete-package"},{"title":"Get Package","api":"GET /agents/packages/{package_id}","tag":"v2.0.0+","href":"/api-reference/agents/packages/get-package"},{"title":"List Packages","api":"GET /agents/packages","tag":"v2.0.0+","href":"/api-reference/agents/packages/list-packages"}]},{"group":"Sessions","pages":[{"title":"Cancel Session","api":"POST /api/agents/sessions/{session_id}/cancel","tag":"v2.0.1+","href":"/api-reference/agents/sessions/cancel-session"},{"title":"Create Session","api":"POST /api/agents/sessions","tag":"v2.0.1+","href":"/api-reference/agents/sessions/create-session"},{"title":"Delete Session","api":"DELETE /api/agents/sessions/{session_id}","tag":"v2.0.1+","href":"/api-reference/agents/sessions/delete-session"},{"title":"Get Session","api":"GET /api/agents/sessions/{session_id}","tag":"v2.0.1+","href":"/api-reference/agents/sessions/get-session"},{"title":"Get Session Results","api":"GET /api/agents/sessions/{session_id}/results","tag":"v2.0.1+","href":"/api-reference/agents/sessions/get-session-results"},{"title":"List Session Events","api":"GET /api/agents/sessions/{session_id}/events","tag":"v2.0.1+","href":"/api-reference/agents/sessions/list-session-events"},{"title":"List Sessions","api":"GET /api/agents/sessions","tag":"v2.0.1+","href":"/api-reference/agents/sessions/list-sessions"},{"title":"Query Session","api":"POST /api/agents/sessions/{session_id}/query","tag":"v2.0.1+","href":"/api-reference/agents/sessions/query-session"}]}]},{"group":"Projects API","root":{"title":"About the Projects API","sidebarTitle":"About","tag":"v2.0.0+","href":"/api-reference/projects/about"},"pages":[{"title":"Create Project","api":"POST /projects","tag":"v2.0.0+","href":"/api-reference/projects/create-project"},{"title":"Delete Project","api":"DELETE /projects/{project_id}","tag":"v2.0.0+","href":"/api-reference/projects/delete-project"},{"title":"Get Project","api":"GET /projects/{project_id}","tag":"v2.0.0+","href":"/api-reference/projects/get-project"},{"title":"List Projects","api":"GET /projects","tag":"v2.0.0+","href":"/api-reference/projects/list-projects"},{"title":"Update Project","api":"PATCH /projects/{project_id}","tag":"v2.0.0+","href":"/api-reference/projects/update-project"}]},{"group":"Inference API","root":{"title":"About the Inference API","sidebarTitle":"About","tag":"v2.0.1+","href":"/api-reference/inference/about"},"pages":[{"title":"Inference: Direct Query","sidebarTitle":"Direct Query","api":"POST /api/inference/query","tag":"v2.0.1+","href":"/api-reference/inference/query"},{"title":"Inference: Chat Completions","sidebarTitle":"Chat Completions","api":"POST /api/inference/chat/completions","tag":"v2.0.1+","href":"/api-reference/inference/chat-completions"},{"title":"Inference: Responses","sidebarTitle":"Responses","api":"POST /api/inference/responses","tag":"v2.0.1+","href":"/api-reference/inference/responses"},{"group":"Sessions","pages":[{"title":"Create Model Session","api":"POST /api/inference/sessions","tag":"v2.0.1+","href":"/api-reference/inference/sessions/create-session"},{"title":"List Model Sessions","api":"GET /api/inference/sessions","tag":"v2.0.1+","href":"/api-reference/inference/sessions/list-sessions"},{"title":"Get Model Session","api":"GET /api/inference/sessions/{session_id}","tag":"v2.0.1+","href":"/api-reference/inference/sessions/get-session"},{"title":"Query a Model Session","api":"POST /api/inference/sessions/{session_id}/query","tag":"v2.0.1+","href":"/api-reference/inference/sessions/query-session"},{"title":"Cancel Model Session","api":"POST /api/inference/sessions/{session_id}/cancel","tag":"v2.0.1+","href":"/api-reference/inference/sessions/cancel-session"},{"title":"Delete Model Session","api":"DELETE /api/inference/sessions/{session_id}","tag":"v2.0.1+","href":"/api-reference/inference/sessions/delete-session"}]},{"title":"Migrating from the Direct Query API","sidebarTitle":"Direct Query API → Inference","tag":"v2.0.1+","href":"/api-reference/inference/migrate-from-direct-query"},{"title":"Migrating Omega from the Direct Query API","sidebarTitle":"Direct Query API → Omega","tag":"v2.0.1+","href":"/api-reference/inference/migrate-omega-from-direct-query"}]},{"group":"Lens API","root":{"title":"About the Lens API","sidebarTitle":"About","href":"/api-reference/lens/about"},"pages":[{"title":"Register Lens","api":"POST /lens/register","tag":"v1.1.5+","href":"/api-reference/lens/lens-register"},{"title":"Clone Lens","api":"POST /lens/clone","tag":"v1.1.5+","href":"/api-reference/lens/lens-clone"},{"title":"Modify Lens","api":"POST /lens/modify","tag":"v1.1.5+","href":"/api-reference/lens/lens-modify"},{"title":"Delete Lens","api":"POST /lens/delete","tag":"v1.1.5+","href":"/api-reference/lens/lens-delete"},{"title":"Get Lens Info","api":"GET /lens/info","tag":"v1.1.5+","href":"/api-reference/lens/get-info"},{"title":"Get Lens Metadata","api":"GET /lens/metadata","tag":"v1.1.5+","href":"/api-reference/lens/get-metadata"},{"title":"Create Lens Session","api":"POST /lens/sessions/create","tag":"v1.1.5+","href":"/api-reference/lens/create-session"},{"title":"Destroy Lens Session","api":"POST /lens/sessions/destroy","tag":"v1.1.5+","href":"/api-reference/lens/destroy-session"},{"title":"Get Sessions Info","api":"GET /lens/sessions/info","tag":"v1.1.5+","href":"/api-reference/lens/get-sessions-info"},{"title":"Get Sessions Metadata","api":"GET /lens/sessions/metadata","tag":"v1.1.5+","href":"/api-reference/lens/get-sessions-metadata"},{"title":"Session Events Process","api":"POST /lens/sessions/events/process","tag":"v1.1.5+","href":"/api-reference/lens/sessions-events-process"},{"title":"Lens Events API (WebSocket)","href":"/api-reference/lens/websocket-events"},{"title":"Consume Session Events (SSE)","api":"GET /lens/sessions/consumer/{session_id}","tag":"v1.1.5+","href":"/api-reference/lens/consumer"},{"title":"Lens API Examples","href":"/api-reference/lens/examples"}]},{"group":"Files API","root":{"title":"About the Files API","sidebarTitle":"About","href":"/api-reference/files/about"},"pages":[{"title":"Upload File","api":"POST /files","tag":"v1.1.5+","href":"/api-reference/files/upload"},{"title":"Upload Base64 File","api":"POST /files/base64","tag":"v1.1.5+","href":"/api-reference/files/upload-base64"},{"title":"Delete File by ID","api":"DELETE /files/{file_id}","tag":"v2.0.0+","href":"/api-reference/files/delete-file-by-id"},{"title":"Delete File","api":"DELETE /files/delete/{file_id}","tag":"v1.1.5+","href":"/api-reference/files/delete-file"},{"title":"Download File Content","api":"GET /files/{file_id}/content","tag":"v2.0.0+","href":"/api-reference/files/download-file-content"},{"title":"Download File","api":"GET /files/download/{file_id}","tag":"v1.1.5+","href":"/api-reference/files/download-file"},{"title":"Get Files Info","api":"GET /files/info","tag":"v1.1.5+","href":"/api-reference/files/get-info"},{"title":"Get File","api":"GET /files/{file_id}","tag":"v2.0.0+","href":"/api-reference/files/get-file"},{"title":"Get File Metadata","api":"GET /files/metadata/{file_id}","tag":"v1.1.5+","href":"/api-reference/files/get-metadata"},{"title":"List Files","api":"GET /files","tag":"v2.0.0+","href":"/api-reference/files/list-files"},{"title":"List Files Metadata","api":"GET /files/metadata","tag":"v1.1.5+","href":"/api-reference/files/get-all-metadata"},{"title":"Initiate Upload","api":"POST /files/uploads/initiate","tag":"v1.1.5+","href":"/api-reference/files/initiate-upload"},{"title":"Generate Upload Part URLs","api":"POST /files/uploads/{upload_id}/parts/urls","tag":"v1.1.5+","href":"/api-reference/files/generate-part-urls"},{"title":"Checkpoint Upload Parts","api":"POST /files/uploads/{upload_id}/parts/checkpoint","tag":"v1.1.5+","href":"/api-reference/files/checkpoint-parts"},{"title":"Complete Upload","api":"POST /files/uploads/{upload_id}/complete","tag":"v1.1.5+","href":"/api-reference/files/complete-upload"},{"title":"Abort Upload","api":"POST /files/uploads/{upload_id}/abort","tag":"v1.1.5+","href":"/api-reference/files/abort-upload"}]},{"group":"Fine-Tuning Service","root":{"title":"About the Fine-Tuning Service API","sidebarTitle":"About","href":"/api-reference/fine-tuning/about"},"pages":[{"title":"Create Fine-tuning Job","sidebarTitle":"Create Job","api":"POST /fine-tuning/jobs","tag":"v1.1.5+","href":"/api-reference/fine-tuning/create-job"},{"title":"Get Fine-tuning Job Details","sidebarTitle":"Get Job Details","api":"GET /fine-tuning/jobs/{id}","tag":"v1.1.5+","href":"/api-reference/fine-tuning/get-job"},{"title":"List Fine-tuning Jobs","sidebarTitle":"List Jobs","api":"GET /fine-tuning/jobs","tag":"v1.1.5+","href":"/api-reference/fine-tuning/list-jobs"},{"title":"Pause Fine-tuning Job","sidebarTitle":"Pause Job","api":"POST /fine-tuning/jobs/{id}/pause","tag":"v1.1.5+","href":"/api-reference/fine-tuning/pause-job"},{"title":"Resume Fine-tuning Job","sidebarTitle":"Resume Job","api":"POST /fine-tuning/jobs/{id}/resume","tag":"v1.1.5+","href":"/api-reference/fine-tuning/resume-job"},{"title":"Cancel Fine-tuning Job","sidebarTitle":"Cancel Job","api":"POST /fine-tuning/jobs/{id}/cancel","tag":"v1.1.5+","href":"/api-reference/fine-tuning/cancel-job"},{"title":"List Job Checkpoints","api":"GET /fine-tuning/jobs/{id}/checkpoints","tag":"v1.1.5+","href":"/api-reference/fine-tuning/list-job-checkpoints"},{"title":"List Job Events","api":"GET /fine-tuning/jobs/{id}/events","tag":"v1.1.5+","href":"/api-reference/fine-tuning/list-job-events"},{"title":"List Job Logs","api":"GET /fine-tuning/jobs/{id}/logs","tag":"v1.1.5+","href":"/api-reference/fine-tuning/list-job-logs"},{"title":"List Job Metrics","api":"GET /fine-tuning/jobs/{id}/metrics","tag":"v1.1.5+","href":"/api-reference/fine-tuning/list-job-metrics"},{"title":"List Checkpoint Options","api":"GET /fine-tuning/checkpoints","tag":"v1.1.5+","href":"/api-reference/fine-tuning/list-checkpoint-options"}]},{"group":"Batch Processing API","root":{"title":"About the Batch Processing API","sidebarTitle":"About","href":"/api-reference/batch/about"},"pages":[{"group":"Jobs","pages":[{"title":"Create Job","api":"POST /batch/jobs","tag":"v1.1.0+","href":"/api-reference/batch/jobs/create-job"},{"title":"List Jobs","api":"GET /batch/jobs","tag":"v1.1.0+","href":"/api-reference/batch/jobs/list-jobs"},{"title":"Get Job","api":"GET /batch/jobs/{id}","tag":"v1.1.0+","href":"/api-reference/batch/jobs/get-job"},{"title":"Retry Job","api":"POST /batch/jobs/{id}/retry","tag":"v1.1.0+","href":"/api-reference/batch/jobs/retry-job"},{"title":"Pause Job","api":"POST /batch/jobs/{id}/pause","tag":"v1.1.5+","href":"/api-reference/batch/jobs/pause-job"},{"title":"Resume Job","api":"POST /batch/jobs/{id}/resume","tag":"v1.1.5+","href":"/api-reference/batch/jobs/resume-job"},{"title":"Delete Job","api":"DELETE /batch/jobs/{id}","tag":"v1.1.0+","href":"/api-reference/batch/jobs/delete-job"},{"title":"Cancel Job","api":"POST /batch/jobs/{id}/cancel","tag":"v1.1.0+","href":"/api-reference/batch/jobs/cancel-job"},{"title":"Get Queue Depths","api":"GET /batch/jobs/queue","tag":"v1.1.0+","href":"/api-reference/batch/jobs/get-queue-depths"}]},{"group":"Events","pages":[{"title":"List Job Events","api":"GET /batch/jobs/{id}/events","tag":"v1.1.0+","href":"/api-reference/batch/events/list-job-events"}]},{"group":"I/O","pages":[{"title":"List Inputs","api":"GET /batch/jobs/{id}/inputs","tag":"v1.1.0+","href":"/api-reference/batch/io/list-inputs"},{"title":"List Outputs","api":"GET /batch/jobs/{id}/outputs","tag":"v1.1.0+","href":"/api-reference/batch/io/list-outputs"},{"title":"List Inputs With Progress","api":"GET /batch/jobs/{id}/inputs/progress","tag":"v1.1.0+","href":"/api-reference/batch/io/list-inputs-progress"},{"title":"Get Inputs Progress Counts","api":"GET /batch/jobs/{id}/inputs/progress/counts","tag":"v1.1.0+","href":"/api-reference/batch/io/get-inputs-progress-counts"},{"title":"Get Inputs Progress Traces","api":"GET /batch/jobs/{id}/inputs/progress/traces","tag":"v1.1.0+","href":"/api-reference/batch/io/get-inputs-progress-traces"},{"title":"Validate Prospective Inputs","api":"POST /batch/jobs/inputs/validation","tag":"v1.1.5+","href":"/api-reference/batch/io/validate-inputs"}]},{"group":"Progress","pages":[{"title":"List Job Progress","api":"GET /batch/jobs/{id}/progress","tag":"v1.1.0+","href":"/api-reference/batch/progress/list-job-progress"}]},{"group":"Registry","pages":[{"title":"List Pipelines","api":"GET /batch/registry/pipelines","tag":"v1.1.0+","href":"/api-reference/batch/registry/pipelines/list-pipelines"},{"title":"Get Pipeline","api":"GET /batch/registry/pipelines/{id}","tag":"v1.1.0+","href":"/api-reference/batch/registry/pipelines/get-pipeline"},{"title":"Get Pipeline Schema","api":"GET /batch/registry/pipelines/{id}/schema","tag":"v1.1.0+","href":"/api-reference/batch/registry/pipelines/get-pipeline-schema"}]}]}]}]},{"tab":"For Agents","pages":[{"title":"For Agents","href":"/for-agents/agent-skills"}]},{"tab":"Admin","pages":[{"title":"Admin","href":"/developer-console/org-admin/overview"}]}]},"basePath":"","title":"Create Eval - Archetype AI Documentation","deploymentId":"dpl_vSEmv961Ncyvz1psVwgUnTXQMSYk","buildId":"dpl_vSEmv961Ncyvz1psVwgUnTXQMSYk","dynamics":["c2aa737661","2b790eda92","b0f84f53ff"],"chunks":["OpenApiReferenceProvider-CrDfRf61.js","snippetPresets-Dv-cOUrx.js","utils-BUTreAgv.js","style-to-object-BY3JEAdk.js","utils-wsznt9RM.js","v4-DIgz8zy6.js","addUniqueKeysToSchema-DEsMlaPy.js","conditional-requirements-CFM0id0Q.js","cloneDeep-DontW4xJ.js","_Uint8Array-pP7X6_RM.js","isObject-FP1Yky1u.js","ApiExamples-CU_aGGP0.js","usePlaygroundInputsStore-7dNmUIbS.js","shallow-fTtruD8x.js","isEqual-BRfNFpSD.js","CodeSnippets-52884wUJ.js","CodeSnippet-eoRjC4Le.js","getApiPlaygroundDisplay-CZyAPlR2.js","Page-Cr9IwkK7.js","DynamicMarkdown-NFqqduU7.js","SectionHeading-DQ-mJoAt.js","icon-DXTBUt6s.js","MethodPill-2v-b3js5.js","getPastedJson-v_8Y1_4v.js","DuplicateIcon-DXIkhZIo.js","createIcon-DRpYuPZs.js","axios-CsQoIw0U.js"]}