At this time, only the Operational State Monitoring (OSM)
blueprint is supported by the Optimizations and Evals APIs.
For a working example of these APIs in use, see our Operational State Monitoring on washing
machines
(LARCO) - Quickstart
demo.
Optimization vs. Fine-Tuning
The optimization process doesn’t perform fine-tuning; rather, it relies on an existing model and simply finds the best configuration of that model for your needs. Fine-tuning is an additional layer of tuning which is is heavier and more in-depth than what optimize does. Fine-tuning is also more generic, since one fine-tuned model can be used by multiple different agents, while the optimize and eval lifecycle is about creating one custom agent.Using the Optimizations API
1
Choose a canonical agent
First, select one of the canonical
agent blueprints to serve as your base. This
is the template for what you’re trying to do.
At this time, only the Operational State Monitoring
blueprint is supported.
2
Start optimization
Run an optimization using the selected blueprint and your labeled training data.
The optimization process generates a series of trials using different configurations,
each with a score to help you select the best one for your data. Use the
Create Optimization endpoint
to create the optimization.
3
Monitor the optimization
Poll the optimization using Get Optimization
until its
status is completed,
or indicates cancellation or failure.4
Select the best trial to promote to a blueprint
There are two ways to choose the best trial. The simplest way is to accept the trial ID
specified by the
best_trial_id
property returned in the results from the Get Optimization endpoint.The other way is to use the List Optimization Trials
endpoint to get the list of trials generated by the optimization progress, then choose the top N of
them (where N is whatever number of trials you want to consider). Then use the Evals API to determine
which of the selected trials is best for your data.Using the Evals API
Once you’ve determined the best trial or trials, you may want to test them to verify their quality. To do so, you need to run an eval pass on one or more of the top-scoring trials using a separate set of data, to confirm which one generates the best results. This is useful because by default the optimization process can “over-fit” on the data you gave it. The eval process is a second check to make sure that the agent you will get using the configured blueprint will work well for you. Do the following for each trial you want to evaluate:1
Promote the trial to a blueprint
Use the Promote Trial endpoint to
create a new custom blueprint from the trial configuration.
2
Create a new eval for the custom blueprint
Use the Create Eval endpoint to create the
new eval. Specify the promoted blueprint’s ID as the
blueprint_id parameter, and
provide the example data set in examples.3
Monitor the eval
Use the Get Eval endpoint periodically to
determine if the eval has reached a terminal state. If the reported
status is
completed, the eval completed successfully. If the status is cancelled or
failed, the eval was cancelled or failed with an error, respectively.4
Examine the results
Use the returned metrics
report to determine
the quality of each configuration. When you find one that meets or exceeds your minimum
threshold, you’ve found a good custom blueprint for your use case.
Using the Custom Blueprint
1
Create a bundle
Once you’ve got your optimized, promoted, and optionally evaluated blueprint, you can
use the Create Bundle endpoint to create
an agent bundle, ready to be run.
2
Run your agent
Run the customized bundle to create a new agent using the Run
Bundle endpoint, just like you would with any
agent bundle.