> ## Documentation Index
> Fetch the complete documentation index at: https://docs.archetypeai.app/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Start with /introduction/getting-started. Use the Inference API's Direct Query endpoint (POST /api/inference/query) with the newton-c-2.6 or newton-fusion-1.0 model (text, image, and video reasoning) or the omega-embedding encoder (time-series embeddings). ATAI_API_ENDPOINT is the base URL https://platform.u1.archetypeai.app with no version path; every endpoint path begins with /api. Pages whose descriptions are marked (Archived) document the legacy Lens runtime — do not use them for new projects.

# Developer Quickstart: Inference API

> Run your first Inference API calls against Newton C 2.6 and Newton Omega 1.5 — describe a short assembly video and detect a machine fault from real sensor data, in a few lines of Python.

This page runs the same kind of examples as [Getting Started: Direct Query API](https://docs.archetypeai.app/introduction/getting-started.md) on the
[Inference API](https://docs.archetypeai.app/api-reference/inference/about), which serves Newton C 2.6, Newton Fusion 1.0 and
Newton Omega 1.5 by name. That page uses the [Direct Query API](https://docs.archetypeai.app/api-reference/query.md), which is
still available.

> [!WARNING]
>
> As of November 15, 2026, some inputs work only on the Direct Query API: still images on
> `/inference/query`, uploaded files, full-resolution frames, and more than 16 video frames. If you
> need one of them, start with [Getting Started: Direct Query API](https://docs.archetypeai.app/introduction/getting-started.md) instead. See
> [Current Limitations](https://docs.archetypeai.app/api-reference/inference/migrate-from-direct-query.md#current-limitations).

## Before you Begin

You'll need Python 3.11 or later and an API key from a production account: go to the API Keys page
(e.g. [platform.u1.archetypeai.app/console/profile](https://platform.u1.archetypeai.app/console/profile))
to create one.

> [!WARNING]
>
> The examples on this page need a production account: the free trial doesn't serve the Newton and
> Omega models they call. On a free trial, start with one of these agent examples instead (see
> [Example Projects](https://docs.archetypeai.app/code-examples/example-projects.md#time-series)):
>
> * [Washing-Machine States](https://github.com/archetypeai/operational-state-monitoring-agent-example-larco-quickstart), with Operational State Monitoring
> * [Chemical-Plant Anomalies](https://github.com/archetypeai/anomaly-discovery-agent-example-tep-quickstart), with Anomaly Discovery
> * [Water-Pump Breakdowns](https://github.com/archetypeai/rare-event-detection-agent-example-pump-quickstart), with Rare Event Detection
> * [Screwdriving Anomalies](https://github.com/archetypeai/omega-encoder-example-aursad-quickstart), with the Omega Encoder agent
> * [Air-Compressor Leaks](https://github.com/archetypeai/label-free-anomaly-discovery-agent-example-metropt-quickstart), with Label-Free Anomaly Discovery

## 1. Setup

Create a virtual environment, install the Archetype AI Python client, and set your credentials.
Replace `"insert_your_key"` with your API key.

```bash theme={"system"}
python3 -m venv venv
source venv/bin/activate

pip install archetypeai
export ATAI_API_KEY="insert_your_key"
export ATAI_API_ENDPOINT="https://platform.u1.archetypeai.app/api"
```

> [!NOTE]
>
> The virtual environment matters: recent Python installs on macOS (Homebrew) and Debian/Ubuntu block system-wide installs, so a bare `pip install` fails with `externally-managed-environment`. If you see that error, activate the venv first. Conda environments work too — see [Python Client Setup](https://docs.archetypeai.app/libraries/python.md).

Both examples below call the Inference API. The `model` field names the model:

* **Newton C 2.6** (`newton-c-2.6`) on `POST /inference/query` — reasons over text and video, and answers in natural language. **Newton Fusion 1.0** (`newton-fusion-1.0`), a larger reasoning model, takes the same requests.
* **Newton Omega 1.5** (`omega-embedding`) — encodes a window of time-series sensor readings into a fixed-size embedding vector per channel, for building classifiers and anomaly detectors. The example opens a session to set its window length, then sends one `POST` per window to that session.

The first call after a model has been idle can take about 40 seconds while a copy of it starts;
later calls take a few seconds.

## 2. Describe a Video with Newton C 2.6

This example sends a short assembly video to Newton C 2.6 and asks what the worker does. The video
goes inline in the request, base64-encoded in `file_ids`: no upload step.

Download the sample clip (470 KB, 31 seconds: a worker fitting an o-ring, a cap and a manifold, then
tightening with a wrench):

```bash macOS / Linux theme={"system"}
curl -sL -o assembly.mp4 "https://raw.githubusercontent.com/archetypeai/agent-skills/main/skills/atai-newton-fusion-model/references/sample_assets/1_pass_2_pass_3_pass_B_small.mp4"
```

```powershell Windows theme={"system"}
curl.exe -sL -o assembly.mp4 "https://raw.githubusercontent.com/archetypeai/agent-skills/main/skills/atai-newton-fusion-model/references/sample_assets/1_pass_2_pass_3_pass_B_small.mp4"
```

> [!NOTE]
>
> On Windows, use `curl.exe` — in PowerShell, plain `curl` is an alias for `Invoke-WebRequest`, which rejects these flags.

Save the following as `video_example.py`:

```python video_example.py expandable theme={"system"}
import base64
import json
import os
import re
import sys

from archetypeai.api_client import ArchetypeAI

# Newton C 2.6. Swap in "newton-fusion-1.0" to try Newton Fusion 1.0.
MODEL = "newton-c-2.6"


def make_client():
    endpoint = os.environ.get(
        "ATAI_API_ENDPOINT", "https://platform.u1.archetypeai.app/api").rstrip("/")
    return ArchetypeAI(os.environ["ATAI_API_KEY"], api_endpoint=endpoint)


def main():
    client = make_client()

    # Describe the sample clip, or pass your own short .mp4 as an argument:
    #   python video_example.py your_video.mp4
    video_path = sys.argv[1] if len(sys.argv) > 1 else "assembly.mp4"

    # The video goes inline: the request body must stay under 2 MiB, so keep
    # the clip under about 1.4 MB.
    with open(video_path, "rb") as video_file:
        video = base64.b64encode(video_file.read()).decode()

    query = "Describe what the person does, step by step."
    if MODEL == "newton-fusion-1.0":
        query += " /no_think"  # Fusion is a reasoning model; skip its thinking step

    body = {
        "model": MODEL,
        "query": query,
        "system_prompt": "You are a quality inspector on an assembly line.",
        "file_ids": [f"base64:{video}"],
        "max_new_tokens": 300,
    }
    result = client.requests_post(
        f"{client.api_endpoint}/inference/query",
        data_payload=json.dumps(body),
        additional_headers={"Content-Type": "application/json"},
    )

    # The model's answer is at response.response[0]. Fusion prefixes it with
    # a <think>...</think> block, which we drop.
    answer = result["response"]["response"][0]
    print(re.sub(r"^\s*<think>.*?</think>\s*", "", answer, flags=re.S))


if __name__ == "__main__":
    main()
```

Then run it:

```bash theme={"system"}
python video_example.py
```

### What you should see

After about 2 seconds (longer on the first call), the model's description prints to your terminal:

```text theme={"system"}
The person picks up a blue cap and examines it. They then pick up a black tool and insert the blue
cap into it. Next, they pick up a wrench and use it to tighten the cap. Finally, they set the tool
down and examine the cap again.
```

> [!NOTE]
>
> The model reads 16 evenly spaced frames from every clip, each resized to 224 × 224 pixels, so a
> brief action or a small detail can be missed. Trim a clip to the action and keep it short.
> To send a larger video, shrink it first, for example
> `ffmpeg -i input.mp4 -vf "scale=448:-2,fps=4" -c:v libx264 -crf 28 -an small.mp4`.

## 3. Detect a Machine Fault with Newton Omega 1.5

The Omega model turns a window of time-series sensor readings into a 768-dimensional embedding — no prompt needed. You send the raw numbers and get vectors back, ready for downstream classification or anomaly detection.

This example runs the full cycle on real vibration data from a bearing run-to-failure experiment:

1. **Build an n-shot library** — embed a handful of labeled windows (healthy and degraded).
2. **Classify held-out windows** the library has never seen, with a k-nearest-neighbors vote over the embeddings.
3. **Score the predictions** against ground-truth labels.

Download the sample data (a \~54 MB subset of the NASA IMS Bearing Dataset — see the attribution note below):

```bash macOS / Linux theme={"system"}
curl -sL --remote-name-all "https://raw.githubusercontent.com/archetypeai/agent-skills/main/skills/atai-newton-omega-model/references/sample_data/bearing_{healthy,degraded,inference,labels}.csv"
```

```powershell Windows theme={"system"}
curl.exe -sL --remote-name-all "https://raw.githubusercontent.com/archetypeai/agent-skills/main/skills/atai-newton-omega-model/references/sample_data/bearing_{healthy,degraded,inference,labels}.csv"
```

Save the following as `omega_example.py`:

```python omega_example.py expandable theme={"system"}
import csv
import json
import math
import os
from concurrent.futures import ThreadPoolExecutor

from archetypeai.api_client import ArchetypeAI

MODEL = "omega-embedding"  # Newton Omega 1.5 on the Inference API
WINDOW = 64        # timesteps per window: any length from 16 to 1024
CHANNEL = "bearing_1"  # the bearing that fails in this experiment
TEST_WINDOWS = 100 # held-out windows to classify (~1000 available)


def make_client():
    endpoint = os.environ.get(
        "ATAI_API_ENDPOINT", "https://platform.u1.archetypeai.app/api").rstrip("/")
    return ArchetypeAI(os.environ["ATAI_API_KEY"], api_endpoint=endpoint)


def read_column(csv_path, column, cast=float):
    """Read one column from a CSV into a list."""
    with open(csv_path, newline="") as csv_file:
        reader = csv.reader(csv_file)
        index = next(reader).index(column)
        return [cast(row[index]) for row in reader]


def open_session(client, window):
    """Open an Inference API session whose window length is `window` timesteps."""
    session = client.requests_post(
        f"{client.api_endpoint}/inference/sessions",
        data_payload=json.dumps({
            "model": MODEL,
            "values": {"window_size": window, "step_size": window},
        }),
        additional_headers={"Content-Type": "application/json"},
    )
    return session["id"]


def close_session(client, session_id):
    """Cancel the session to free its place under your organization's limit."""
    client.requests_post(
        f"{client.api_endpoint}/inference/sessions/{session_id}/cancel",
        data_payload=b"")


def embed(client, session_id, values):
    """Embed one WINDOW-timestep window into a 768-d vector."""
    body = {
        "model": MODEL,
        "query": "",
        "events": [
            {"type": "data.numeric_array", "event_data": {"contents": [values]}}
        ],
    }
    result = client.requests_post(
        f"{client.api_endpoint}/inference/sessions/{session_id}/query",
        data_payload=json.dumps(body),
        additional_headers={"Content-Type": "application/json"},
    )
    answer = result["response"]
    # A window shorter than the session's window length returns success=false
    # with an empty response, not an HTTP error.
    if not answer.get("success"):
        raise RuntimeError(f"embedding failed: {answer}")
    return answer["response"][0]  # one row per window: 768 values per channel


def knn_predict(embedding, library, k=3):
    """Majority vote among the k nearest labeled library embeddings."""
    distances = sorted(
        (math.dist(embedding, lib_embedding), label)
        for lib_embedding, label in library
    )
    votes = [label for _, label in distances[:k]]
    return max(set(votes), key=votes.count)


def main():
    client = make_client()
    session_id = open_session(client, WINDOW)
    try:
        classify(client, session_id)
    finally:
        close_session(client, session_id)


def classify(client, session_id):
    # 1. Build a small n-shot library from the labeled shot files.
    #    Scale every window with ONE scaler fit on the library pool, so the
    #    library and the test windows share the same reference frame.
    healthy = read_column("bearing_healthy.csv", CHANNEL)
    degraded = read_column("bearing_degraded.csv", CHANNEL)
    pool = healthy + degraded
    mean = sum(pool) / len(pool)
    std = (sum((value - mean) ** 2 for value in pool) / len(pool)) ** 0.5

    def scale(values):
        return [(value - mean) / std for value in values]

    starts = range(0, len(healthy) - WINDOW + 1, 256)  # overlapping shots
    shots = [(scale(healthy[s:s + WINDOW]), "healthy") for s in starts]
    shots += [(scale(degraded[s:s + WINDOW]), "degraded") for s in starts]

    with ThreadPoolExecutor(max_workers=8) as pool_executor:
        vectors = list(pool_executor.map(
            lambda shot: embed(client, session_id, shot[0]), shots))
    library = [(vector, label) for vector, (_, label) in zip(vectors, shots)]
    print(f"n-shot library: {len(library)} labeled windows")

    # 2. Classify held-out windows the library has never seen.
    #    bearing_inference.csv has no label column, so leakage is impossible.
    timestamps = read_column("bearing_inference.csv", "timestamp", cast=int)
    series = read_column("bearing_inference.csv", CHANNEL)

    # Non-overlapping windows that don't span a gap in the recording,
    # evenly spread across the timeline.
    all_starts = [
        start for start in range(0, len(series) - WINDOW + 1, WINDOW)
        if timestamps[start + WINDOW - 1] - timestamps[start] == WINDOW - 1
    ]
    stride = max(1, len(all_starts) // TEST_WINDOWS)
    test_starts = all_starts[::stride][:TEST_WINDOWS]

    with ThreadPoolExecutor(max_workers=8) as pool_executor:
        predictions = list(pool_executor.map(
            lambda start: knn_predict(
                embed(client, session_id, scale(series[start:start + WINDOW])), library),
            test_starts))

    # 3. Score against the ground-truth labels (used ONLY for scoring).
    truth_by_timestamp = dict(zip(
        read_column("bearing_labels.csv", "timestamp", cast=int),
        read_column("bearing_labels.csv", "label", cast=str)))
    truths = [truth_by_timestamp[timestamps[start]] for start in test_starts]

    for start, pred, truth in zip(test_starts, predictions, truths):
        marker = "ok" if pred == truth else "MISS"
        print(f"  ts={timestamps[start]:>8}  predicted={pred:<8} truth={truth:<8} {marker}")

    # Metrics with "degraded" as the positive class: did we catch the fault?
    pairs = list(zip(predictions, truths))
    correct = sum(pred == truth for pred, truth in pairs)
    true_pos = sum(pred == truth == "degraded" for pred, truth in pairs)
    true_neg = sum(pred == truth == "healthy" for pred, truth in pairs)
    false_pos = sum(pred == "degraded" and truth == "healthy" for pred, truth in pairs)
    false_neg = sum(pred == "healthy" and truth == "degraded" for pred, truth in pairs)
    precision = true_pos / (true_pos + false_pos) if true_pos + false_pos else 0.0
    recall = true_pos / (true_pos + false_neg) if true_pos + false_neg else 0.0
    f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
    print("confusion matrix (rows: truth, cols: predicted)")
    print(f"{'':>16}{'healthy':>9}{'degraded':>10}")
    print(f"{'healthy':>16}{true_neg:>9}{false_pos:>10}")
    print(f"{'degraded':>16}{false_neg:>9}{true_pos:>10}")
    print(f"accuracy : {correct}/{len(pairs)} = {correct / len(pairs):.0%}")
    print(f"precision: {precision:.3f}  recall: {recall:.3f}  f1: {f1:.3f}"
          f"  (positive class: degraded)")


if __name__ == "__main__":
    main()
```

Run it:

```bash theme={"system"}
python omega_example.py
```

### What you should see

The 116 embedding calls (16 library windows, 100 test windows) fan out 8-way in parallel; the run takes about 10 seconds:

```text theme={"system"}
n-shot library: 16 labeled windows
  ts= 3000000  predicted=healthy  truth=healthy  ok
  ts= 3010368  predicted=healthy  truth=healthy  ok
  ts= 3020736  predicted=healthy  truth=healthy  ok
  ...
  ts=18996064  predicted=degraded truth=degraded ok
  ts=19006432  predicted=degraded truth=degraded ok
confusion matrix (rows: truth, cols: predicted)
                  healthy  degraded
         healthy       47         4
        degraded        2        47
accuracy : 94/100 = 94%
precision: 0.922  recall: 0.959  f1: 0.940  (positive class: degraded)
```

94 of 100 held-out windows classified correctly from 16 labeled examples — no model training, no feature engineering. The [`atai-newton-omega-model` skill](https://github.com/archetypeai/agent-skills/tree/main/skills/atai-newton-omega-model) runs this same kind of pipeline at full scale.

### Choosing the Window Length

`WINDOW` sets the session's `window_size`, and the script uses it for both the library and the test windows. The right length depends on the signal, so compare a few. On this data, with the settings above:

| `WINDOW` | Library windows | Accuracy | Precision | Recall |
| - | - | - | - | - |
| 64 | 16 | 94% | 0.92 | 0.96 |
| 256 | 14 | 85% | 0.77 | 0.98 |
| 1024 | 8 | 82% | 0.73 | 1.00 |

Every length catches nearly all degraded windows. Longer windows flag more healthy windows as degraded, so precision drops as the window grows. Don't assume 1024 is best: pick the length from a held-out comparison like this one.

> [!NOTE]
>
> The session fixes the window length: every request sent to it is cut into `window_size`-timestep windows, and each full window returns one row of 768 values per channel. A shorter input returns `success: false` with an empty response, and leftover samples past the last full window are dropped. Windows shorter than 1024 are padded and masked inside the model. Omega normalizes each channel inside every window and still sees its overall level, so there is no `normalize_input` option to set.
>
> A session ends when you cancel it, when it sits idle long enough, or when the server holding it is replaced; a query on an ended session returns HTTP 409, and the fix is to open a new one. An organization can keep up to 16 sessions open on a model. Without a session, `POST /inference/query` uses a fixed 1024-timestep window.

> [!NOTE]
>
> **Data attribution:** the sample files are a curated subset of the **NASA IMS Bearing Dataset** (Set 2: a bearing run-to-failure experiment with four accelerometer channels), produced by the University of Cincinnati Center for Intelligent Maintenance Systems (IMS) and distributed via the [NASA Prognostics Data Repository](https://www.nasa.gov/intelligent-systems-division/discovery-and-systems-health/pcoe/pcoe-data-set-repository/). If you use this data in a publication or redistribute it, include NASA's requested citation: J. Lee, H. Qiu, G. Yu, J. Lin, and Rexnord Technical Services (2007). *Bearing Data Set*, IMS, University of Cincinnati. NASA Prognostics Data Repository, NASA Ames Research Center, Moffett Field, CA.

## Try Variations

* **Try Newton Fusion 1.0** — set `MODEL = "newton-fusion-1.0"` in `video_example.py`. It's a reasoning model: the script adds `/no_think` for a direct answer; remove it to let the model reason first, and raise `max_new_tokens` to at least `2048`. See [Reasoning](https://docs.archetypeai.app/api-reference/inference/migrate-from-direct-query.md#reasoning).
* **Ask for structured output** — make the prompt the schema, for example: `Respond with ONLY a JSON object: {"steps": ["<step>", ...], "tool_used": "<tool>"}`.
* **Send an image** — images go to the OpenAI-compatible [Chat Completions](https://docs.archetypeai.app/api-reference/inference/chat-completions.md) endpoint as a data URI, not to `/inference/query`. See [Images](https://docs.archetypeai.app/api-reference/inference/migrate-from-direct-query.md#images).
* **Vary the Omega window length** — change `WINDOW` to 32, 128 or 512 and compare with the [table above](#choosing-the-window-length); the script opens its session with that length.
* **Evaluate more held-out windows** — raise `TEST_WINDOWS` from 100 toward the \~1000 available; each window is one independent call to the session.

## Build with a Coding Agent

The official [Agent Skills](https://docs.archetypeai.app/for-agents/agent-skills.md) teach coding agents like Claude Code, Cursor, and Codex these patterns — including the gotchas — so you can describe what you want and let the agent write the code.

## Troubleshooting

If you're getting errors, see the [Troubleshooting](https://docs.archetypeai.app/support/troubleshooting.md) page, or the errors you're most likely to meet on the Inference API, listed in [Migrating from the Direct Query API](https://docs.archetypeai.app/api-reference/inference/migrate-from-direct-query.md#errors) and [Migrating Omega from the Direct Query API](https://docs.archetypeai.app/api-reference/inference/migrate-omega-from-direct-query.md#errors). A `400` with `model '<model>' refused this request` usually means an input or field the Inference API doesn't support yet; see [Current Limitations](https://docs.archetypeai.app/api-reference/inference/migrate-from-direct-query.md#current-limitations).

## Learn More

> [card] [Inference API](https://docs.archetypeai.app/api-reference/inference/about)
>
> The endpoints, the models they serve, sessions, and errors.

> [card] [Migrating from the Direct Query API](https://docs.archetypeai.app/api-reference/inference/migrate-from-direct-query.md)
>
> Moving Newton C 2.6 calls to the Inference API, and when to stay on the Direct Query API.

> [card] [Migrating Omega from the Direct Query API](https://docs.archetypeai.app/api-reference/inference/migrate-omega-from-direct-query.md)
>
> Moving from Omega 1.4 to Omega 1.5 embeddings.

> [card] [Example Projects](https://docs.archetypeai.app/code-examples/example-projects.md#inference-api)
>
> Working demos on the Inference API, from traffic analysis to wind-turbine fault detection.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.