Fine-tuning is a process by which you train one of Archetype’s models with a data set specifically designed to optimize the model’s performance for your use case. This is the most powerful customization option, producing a model specifically adapted to your sensor configuration or use case. The result of fine-tuning Newton is a forked instance of Newton configured for your use case, encrypted and access-controlled with visibility to your organization only.Documentation Index
Fetch the complete documentation index at: https://docs.archetypeai.app/llms.txt
Use this file to discover all available pages before exploring further.
When to Use Fine-Tuning
Fine-tuning is appropriate when:- You have a unique use case or bespoke sensor that can benefit from custom training on a proprietary dataset
- You have a substantial labeled dataset (typically hundreds or thousands of examples)
- N-shot examples aren’t sufficient for your accuracy requirements
- You need Newton to learn complex patterns specific to your domain
What Fine-Tuning Produces
The result of fine-tuning is a new Newton instance with:- A custom model and inference configuration, tuned for your use case
- Encryption and access controls limiting visibility to your organization only
- Behavior adapted to your specific sensor configuration and use case based on the knowledge gained from fine tuning on your custom dataset.
Labeled Examples
Fine-tuning relies on high-quality examples to help steer a pre-trained instance of Newton to a new sensor configuration or use case. Each labeled example consists of an input and output pair in JSON or YAML format. This provides Newton the information required to understand that given input X, it should generate output Y. As Newton is a multimodal model, the input (X) can consist of one or more sensor types, with the output (Y) currently constrained to text output.Fine-Tuning Using the Fine-Tuning Node
To perform fine-tuning, use the Fine-Tuning Node to run a fine-tuning job. This job evaluates the example datasets, generating the custom model. Once you have a satisfactory model, work with your Archetype AI representative to enable its use.Fine-Tuning Job Lifecycle
When you start a fine-tuning job using the Create Job endpoint, the job begins in theREGISTERED state. When the job starts running, it transitions
first to the STARTING state, then into the RUNNING state.
While the job is running, poll its status using the Get Job
endpoint. When the returned status is COMPLETED, the job has finished running and the
generated model can be found at the path specified by the inference_config_path property of
the returned job record.
To cancel the job immediately without finalizing it, use the Cancel
Job endpoint. Canceling a job means no further checkpoints
are written, so that any configuration derived since the last checkpoint is lost. Canceled jobs
end in the CANCELLED state.
To instead stop the job, finalizing it so that the training performed so far is checkpointed,
use the Stop Job endpoint. This causes the job to
transition into the STOPPING state; while in this state, the job writes the training
checkpoints into the new model bundle. Once the job enters the STOPPED state, you can use the
model specified by the inference_config_path property of the record returned by the Get
Job endpoint.
Running a Fine-Tuning Job
- Upload the data files to process using the Files API.
- Start the fine-tuning job running by using the Create Job endpoint.
- Periodically poll the Get Job endpoint. When the returned
job record’s
statusisCOMPLETEDorSTOPPED, you can get the inference configuration file’s AWS S3 URI from theinference_config_pathproperty of the returned job record. If thestatusisCOMPLETED, this configured model is based on all of the data in the example datasets; if thestatusisSTOPPED, the inference configuration is based on partial data recorded following a call to Stop Job. - Contact your Archetype AI representative for their assistance enabling use of the generated model.