> ## Documentation Index
> Fetch the complete documentation index at: https://docs.archetypeai.app/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Start with /introduction/getting-started. Use the Direct Query API (POST /api/query) with the Newton Fusion model (text, image, and video reasoning) or the Newton Omega encoder (time-series embeddings). ATAI_API_ENDPOINT is the base URL https://api.u1.archetypeai.app with no version path; every endpoint path begins with /api. Pages whose descriptions are marked (Archived) document the legacy Lens runtime — do not use them for new projects.

# Optimizations and Evals

> Determine the best configuration to use with your data for a given blueprint, and run test evaluations to confirm the quality of the configuration.

Within the Agents API are sub-APIs that let you customize and evaluate blueprints for use with
your data.

The Optimizations API accepts a canonical blueprint, your labeled training data, and a search
space (window size, kNN metric, k) and it runs trials. Each trial fits and scores its own
classifier. Pick the resulting trial that best works with your data and promote it to be a
custom blueprint.

The Evals API scores a given trial one time, using a set of labeled data that was not used for
tuning. This gives you an honest quality measurement before you choose to deploy your custom
configuration.

<Note>
  At this time, only the [Operational State Monitoring](/core-concepts/agents/osm) (OSM)
  blueprint is supported by the Optimizations and Evals APIs.
</Note>

Together these APIs are a critical part of the process of creating an agent optimized for your
data.

<Note>
  For a working example of these APIs in use, see our [Operational State Monitoring on washing
  machines
  (LARCO) - Quickstart](https://github.com/archetypeai/operational-state-monitoring-agent-example-larco-quickstart)
  demo.
</Note>

## Optimization vs. Fine-Tuning

The optimization process doesn't perform fine-tuning; rather, it relies on an existing
model and simply finds the best configuration of that model for your needs. Fine-tuning is an
additional layer of tuning which is is heavier and more in-depth than what optimize does.
Fine-tuning is also more generic, since one fine-tuned model can be used by multiple different
agents, while the optimize and eval lifecycle is about creating one custom agent.

## Using the Optimizations API

<Steps>
  <Step title="Choose a canonical agent">
    First, select one of the [canonical
    agent blueprints](/core-concepts/agents/overview#canonical-blueprints) to serve as your base. This
    is the template for what you're trying to do.

    <Note>
      At this time, only the [Operational State Monitoring](/core-concepts/agents/osm)
      blueprint is supported.
    </Note>
  </Step>

  <Step title="Start optimization">
    Run an optimization using the selected blueprint and your labeled training data.
    The optimization process generates a series of trials using different configurations,
    each with a score to help you select the best one for your data. Use the
    [Create Optimization](/api-reference/agents/optimizations/create-optimization) endpoint
    to create the optimization.
  </Step>

  <Step title="Monitor the optimization">
    Poll the optimization using [Get Optimization](/api-reference/agents/optimizations/get-optimization)
    until its [`status`](/api-reference/agents/optimizations/get-optimization#param-status) is `completed`,
    or indicates cancellation or failure.
  </Step>

  <Step title="Select the best trial to promote to a blueprint">
    There are two ways to choose the best trial. The simplest way is to accept the trial ID
    specified by the [`best_trial_id`](/api-reference/agents/optimizations/get-optimization#param-best-trial-id)
    property returned in the results from the **Get Optimization** endpoint.

    The other way is to use the [List Optimization Trials](/api-reference/agents/optimizations/list-optimization-trials)
    endpoint to get the list of trials generated by the optimization progress, then choose the top *N* of
    them (where *N* is whatever number of trials you want to consider). Then use the Evals API to determine
    which of the selected trials is best for your data.
  </Step>
</Steps>

## Using the Evals API

Once you've determined the best trial or trials, you may want to test them to verify their
quality. To do so, you need to run an eval pass on one or more of the top-scoring trials using a
separate set of data, to confirm which one generates the best results. This is useful because by
default the optimization process can "over-fit" on the data you gave it.

The eval process is a second check to make sure that the agent you will get using the configured
blueprint will work well for you.

Do the following for each trial you want to evaluate:

<Steps>
  <Step title="Promote the trial to a blueprint">
    Use the [Promote Trial](/api-reference/agents/optimizations/promote-trial) endpoint to
    create a new custom blueprint from the trial configuration.
  </Step>

  <Step title="Create a new eval for the custom blueprint">
    Use the [Create Eval](/api-reference/agents/evals/create-eval) endpoint to create the
    new eval. Specify the promoted blueprint's ID as the `blueprint_id` parameter, and
    provide the example data set in `examples`.
  </Step>

  <Step title="Monitor the eval">
    Use the [Get Eval](/api-reference/agents/evals/get-eval) endpoint periodically to
    determine if the eval has reached a terminal state. If the reported `status` is
    `completed`, the eval completed successfully. If the `status` is `cancelled` or
    `failed`, the eval was cancelled or failed with an error, respectively.
  </Step>

  <Step title="Examine the results">
    Use the returned [metrics
    report](/api-reference/agents/evals/get-eval#metrics-report-metrics_report) to determine
    the quality of each configuration. When you find one that meets or exceeds your minimum
    threshold, you've found a good custom blueprint for your use case.
  </Step>
</Steps>

## Using the Custom Blueprint

<Steps>
  <Step title="Create a bundle">
    Once you've got your optimized, promoted, and optionally evaluated blueprint, you can
    use the [Create Bundle](/api-reference/agents/bundles/create-bundle) endpoint to create
    an agent bundle, ready to be run.
  </Step>

  <Step title="Run your agent">
    Run the customized bundle to create a new agent using the [Run
    Bundle](/api-reference/agents/bundles/run-bundle) endpoint, just like you would with any
    agent bundle.
  </Step>
</Steps>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.