> ## Documentation Index
> Fetch the complete documentation index at: https://docs.archetypeai.app/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Start with /introduction/getting-started. Use the Inference API's Direct Query endpoint (POST /api/inference/query) with the newton-c-2.6 or newton-fusion-1.0 model (text, image, and video reasoning) or the omega-embedding encoder (time-series embeddings). ATAI_API_ENDPOINT is the base URL https://platform.u1.archetypeai.app with no version path; every endpoint path begins with /api. Pages whose descriptions are marked (Archived) document the legacy Lens runtime — do not use them for new projects.

# Migrating from the Direct Query API

> Move Newton C 2.6 calls from /lens/query to the Inference API, choose between Newton C 2.6 and Newton Fusion 1.0, and what to watch for along the way.

Applies to [version 2.0.1](https://docs.archetypeai.app/release-notes/2.0.x.md#v2-0-1) or later of the Archetype platform.

Newton models are moving from the [Direct Query API](https://docs.archetypeai.app/api-reference/query.md) (`POST /lens/query`)
to the [Inference API](https://docs.archetypeai.app/api-reference/inference/about), which runs platform-hosted models by name
and offers two of them for text, image, and video:

* **`newton-c-2.6`**: Newton C 2.6, the model you called on the Direct Query API as
  `Newton::c2_6_8b_fp8_...`. An 8B model that answers directly.
* **`newton-fusion-1.0`**: Newton Fusion 1.0, a larger model that can reason step by step before
  it answers.

[`POST /inference/query`](https://docs.archetypeai.app/api-reference/inference/query) takes the familiar Direct Query
body, so most scripts move with a new URL and a new model name. Images, video, and inline data are
the exception: several ways of attaching them are refused, and the models see video differently.
This page maps the old request to the new one, shows the attachment paths that work, and lists
what changed in practice, from moving our own demos and agent skills.

The Direct Query API is still available, and for some inputs it is still the only option. Check
[Current Limitations](#current-limitations) before you move a script that sends images, uploaded
files, long videos, or asks for long answers.

Moving Omega embedding calls instead? See [Migrating Omega from the Direct Query
API](https://docs.archetypeai.app/api-reference/inference/migrate-omega-from-direct-query.md).

To check that your platform serves the Inference API, send a short text query. A `200` means it
does; a `404` means it doesn't yet.

```bash theme={"system"}
curl -s -o /dev/null -w "%{http_code}\n" -X POST "$ATAI_API_ENDPOINT/inference/query" \
  -H "Authorization: Bearer $ATAI_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "newton-c-2.6", "query": "Say hello.", "max_new_tokens": 10}'
```

## What Changed

| | Direct Query API | Inference API |
| - | - | - |
| Endpoint | `POST /lens/query` | `POST /inference/query`, or the OpenAI-compatible [`/chat/completions`](https://docs.archetypeai.app/api-reference/inference/chat-completions) and [`/responses`](https://docs.archetypeai.app/api-reference/inference/responses) |
| Model | `Newton::c2_6_8b_fp8_260424d7a55d5e` | `newton-c-2.6` or `newton-fusion-1.0`. The old identifier returns `404` |
| Video | An `.mp4` in `file_ids`, with `max_frames` (default 32, up to 64) | An `.mp4` sent inline as `base64:<data>` in `file_ids`. `max_frames` is refused; every clip is read as 16 frames |
| Images | `data.base64_img` events, or uploaded files in `file_ids` by name, with `multi_image` | An image as a data URI on `/chat/completions` (`image_url`) or `/responses` (`input_image`) |
| Inline data | `data.text` and `data.json` events | Put the data in `query`. `data.text` and `data.json` events are refused |
| Input resolution | | Every image and video frame is resized to 224 × 224 pixels |
| Uploaded files | Referenced by name in `file_ids` | Refused, by name or by `fil_` ID. Send media inline |
| Answer | Plain text | Plain text from Newton C 2.6. Newton Fusion 1.0 starts its answer with a `<think>...</think>` block |
| Time limit | | About 60 seconds per query, then `504` |
| First call | | Can take about 40 seconds while a copy of the model starts |

## Current Limitations

> [!WARNING]
>
> As of November 15, 2026, on the production platform. This section will be updated as these
> limitations change.

Some things that work on the Direct Query API don't work on the Inference API yet. If your app
depends on one of them, keep that call on the Direct Query API for now.

| You need | On the Inference API today | Stay on the Direct Query API if |
| - | - | - |
| **Still images on `/inference/query`** | Refused, whether sent as a `data.base64_img` event, as `base64:` in `file_ids`, or as an uploaded file. Images work only as data URIs on `/chat/completions` and `/responses` | You'd rather not split your image calls from your text and video calls |
| **Uploaded files** | Refused in `file_ids` on `/inference/query`, by name or by `fil_` ID, for images and video alike. Media must go inline, in a request body under 2 MiB: about 1.4 MB of video once base64-encoded | You send clips or images larger than that, or files already uploaded to the platform |
| **Full-resolution images and frames** | Every image and video frame is resized to 224 × 224 pixels, on every endpoint. Asking for `detail: "high"` makes no difference | Your task depends on fine detail: small text, distant objects, thin smoke |
| **More than 16 video frames** | Every clip is read as 16 evenly spaced frames; `max_frames` is refused | A brief action in a long clip matters and you can't cut the clip into short pieces |
| **Long answers** | A query is stopped after about 60 seconds (`504`), and answers are generated more slowly than on the Direct Query API. Newton C 2.6 wrote about 40 tokens (100–140 characters) a second against about 210 characters on the Direct Query API, so answers over about 2,000 tokens time out | You need answers longer than a few hundred words and can't split the request |
| **`repetition_penalty` on images** | Applied on `/inference/query`, but ignored on `/chat/completions`, the route images need. A model that repeats itself while listing or counting can't be steered there | Your image prompts trigger repetition loops |
| **Clear errors** | A refused field or input returns `400 model '<model>' refused this request`, without naming what was refused | |

To keep a long answer within the time limit, ask for a shorter one in the prompt (for example,
"Keep the answer under 300 words") and lower `max_new_tokens` to about `1536`.

## Old Fields, New Fields

| Field | On `/inference/query` |
| - | - |
| `model` | `newton-c-2.6` or `newton-fusion-1.0` |
| `query`, `system_prompt`, `instruction_prompt` | Unchanged |
| `max_new_tokens`, `temperature`, `do_sample`, `top_p`, `top_k`, `repetition_penalty` | Unchanged. Newton Fusion 1.0 needs a larger `max_new_tokens` when it reasons; see [Reasoning](#reasoning) |
| `file_ids` | Send video as `"base64:<mp4 data>"`. Uploaded files are refused, by name or by `fil_` ID |
| `events` | Only `data.numeric_array`, for `omega-embedding`; see [Migrating Omega](https://docs.archetypeai.app/api-reference/inference/migrate-omega-from-direct-query.md). Other event types are refused |
| `response_start_prompt` | Accepted. With Newton Fusion 1.0 it returns an empty answer, so leave it out |
| `max_frames`, `multi_image: true`, `presence_penalty`, `template_name`, `render: true` | Refused with `400` |

The response has the same shape as before: the answer is at `response.response[0]`.

## Moving a Script

**1. Change the URL and the model**

Replace `/lens/query` with `/inference/query`, and `Newton::c2_6_8b_fp8_...` with
`newton-c-2.6`. A text-only query now works as before.

```bash theme={"system"}
curl -X POST "$ATAI_API_ENDPOINT/inference/query" \
  -H "Authorization: Bearer $ATAI_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "newton-c-2.6",
    "query": "List three checks to run before starting a conveyor belt.",
    "max_new_tokens": 300
  }'
```

**2. Remove the fields the Inference API refuses**

Drop `max_frames`, `multi_image`, `presence_penalty`, and `template_name`. A refused field
returns `400` with the message `model '<model>' refused this request`, which doesn't name the
field. If you get that message, remove optional fields one at a time to find it.

**3. Move images, video, and data to a path that works**

Video goes inline in `file_ids`, images go to `/chat/completions` or `/responses`, and text or
tabular data goes in the query itself. See [Sending Images, Video, and
Data](#sending-images-video-and-data).

**4. Allow for the first call**

Raise your client timeout to about 180 seconds. The first call after a model has been idle can
take about 40 seconds while a copy of it starts; later calls take a few seconds.

**5. Try Newton Fusion 1.0**

Change the model to `newton-fusion-1.0` and handle its reasoning block. See [Choosing a
Model](#choosing-a-model) and [Reasoning](#reasoning).

## Sending Images, Video, and Data

### Video

Send the clip inline as `base64:` followed by the file's base64 data, in `file_ids`. The model
reads 16 evenly spaced frames from it.

```python analyze_video.py theme={"system"}
import base64
import os
import re
import sys

import requests

MODEL = "newton-c-2.6"  # or "newton-fusion-1.0"

base_url = os.environ["ATAI_API_ENDPOINT"]
api_key = os.environ["ATAI_API_KEY"]

with open(sys.argv[1], "rb") as f:
    clip = base64.b64encode(f.read()).decode()

query = "Describe what the person does, step by step."
if MODEL == "newton-fusion-1.0":
    query += " /no_think"  # answer without a reasoning step

response = requests.post(
    f"{base_url}/inference/query",
    headers={"Authorization": f"Bearer {api_key}"},
    json={
        "model": MODEL,
        "query": query,
        "system_prompt": "You are a quality inspector on an assembly line.",
        "file_ids": [f"base64:{clip}"],
        "max_new_tokens": 300,
    },
    timeout=180,
)
response.raise_for_status()

answer = response.json()["response"]["response"][0]
# Fusion starts its answer with a <think>...</think> block; drop it.
print(re.sub(r"^\s*<think>.*?</think>\s*", "", answer, flags=re.S))
```

Run against a 31-second assembly clip, `newton-c-2.6` answered:

```text theme={"system"}
The person picks up a blue cap and examines it. They then pick up a black tool and insert the
blue cap into it. Next, they pick up a wrench and use it to tighten the cap. Finally, they set
the tool down and examine the cap again.
```

Keep the request body under 2 MiB, which leaves room for about 1.4 MB of video once it is
base64-encoded. A body just over 2 MiB can hang without an answer, and larger ones are refused
with `413`. Because the model only sees 16 frames at 224 × 224, you lose nothing by shrinking the
clip first:

```bash theme={"system"}
ffmpeg -i input.mp4 -vf "scale=448:-2,fps=4" -c:v libx264 -crf 28 -an small.mp4
```

The same clip can go to `/chat/completions` as a `video_url` content part, or to `/responses` as
an `input_video` content part, each with a `data:video/mp4;base64,...` URL.

### Images

Send images as a data URI on the OpenAI-compatible endpoints: an `image_url` content part on
`/chat/completions`, or an `input_image` part on `/responses`. Because these endpoints follow the
OpenAI format, the OpenAI SDK works with your Archetype API key:

```python describe_image.py theme={"system"}
import base64
import os
import sys

from openai import OpenAI

client = OpenAI(
    base_url=f"{os.environ['ATAI_API_ENDPOINT']}/inference",
    api_key=os.environ["ATAI_API_KEY"],
    timeout=180,
)

with open(sys.argv[1], "rb") as f:
    image = base64.b64encode(f.read()).decode()

completion = client.chat.completions.create(
    model="newton-c-2.6",
    max_tokens=200,
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Describe this image in one sentence."},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image}"}},
        ],
    }],
)
print(completion.choices[0].message.content)
```

On `/inference/query`, `data.base64_img` events and `base64:` images in `file_ids` are
refused, and so is `multi_image: true`.

Every image is resized to 224 × 224 pixels, whatever its size or shape, so small text and fine
detail are lost: a 1024-pixel image reads the same as a 224-pixel copy. To read a label, a gauge,
or a part number, crop to it before sending. When it can't read a detail, Newton Fusion 1.0 may
guess, while Newton C 2.6 tends to say it can't tell, so check answers that depend on fine detail.

### Text and Tabular Data

`data.text` and `data.json` events are refused, and so are text files attached in `file_ids`. Put
the data in the query, after the question:

```json theme={"system"}
{
  "model": "newton-c-2.6",
  "query": "What is the highest temperature in this CSV?\nt,temp\n1,20\n2,35\n3,28",
  "max_new_tokens": 50
}
```

## Choosing a Model

Both models take the same inputs and the same request body, so you can switch with one field.

1. **Start with Newton C 2.6.** It is the model you used on the Direct Query API, it answers
   quickly, and it is enough for describing a scene, summarizing a clip, or classifying a state.
2. **Try Newton Fusion 1.0 when the task needs judgement over several steps**, such as checking
   that each step of a procedure was done. Send it one short clip per step rather than one long
   clip. In our test of three assembly steps, Fusion judged all three correctly from one clip
   per step, against none from the full clip; Newton C 2.6 judged one of three correctly either
   way.
3. **Turn on Fusion's reasoning when accuracy matters more than speed.** Reasoning takes longer
   and needs a larger token budget; see [Reasoning](#reasoning).

Whichever model you use, frame the input for what it can see: 16 frames at 224 × 224. Trim a clip
to the action, and crop to the region that matters.

## Reasoning

Newton Fusion 1.0 is a reasoning model. Its answer starts with a `<think>...</think>` block, which
is empty when it answers directly. Remove the block before showing the answer, as the [video
example](#video) does.

* **To answer directly**, end the query with ` /no_think`. This works when the request includes
  an image or video. On a text-only request the model reasons anyway.
* **To let it reason**, send the query without a switch. `/think` doesn't force reasoning; the
  model decides. Set `max_new_tokens` to at least `2048` for images and video and `4096` for
  text, and `repetition_penalty` to `1.1`. With a smaller budget, the answer can run out of
  tokens inside the reasoning block and come back with no answer at all; without the penalty,
  the reasoning can repeat itself, for example while counting vehicles in a busy scene.
* **Expect it to take longer.** On a short video clip, an answer that reasons can take 20 to 50
  seconds, against a few seconds without reasoning. A model has a time budget of about 60
  seconds, after which the request returns `504`.

If an answer has an opening `<think>` and no closing `</think>`, the model ran out of tokens while
reasoning: raise `max_new_tokens`.

## What Changed in Practice

These notes come from moving a live traffic-camera demo and our Newton agent skills to the
Inference API.

### Video Frames

The Direct Query API let you choose how many frames to read from a video, up to 64. The Inference
API reads 16 evenly spaced frames from every clip, at 224 × 224, and refuses `max_frames`. A
brief action in a long clip can fall between frames, so cut long videos into short clips around
the moments you care about, and send one request per clip.

### Long Answers

Text-only questions that ask for an open-ended analysis can run past the time limit. Moving an
earthquake-monitoring demo, the question "Are any of these earthquakes near major population
centers?" returned `504` every time with the Direct Query API's settings (`max_new_tokens: 3072`),
while the Direct Query API answered it in 8 to 50 seconds. Asking for answers under 300 words and
lowering `max_new_tokens` to `1536` brought all 24 test questions in at 8 to 23 seconds. Listing numbers,
Newton C 2.6 sometimes repeats itself ("M1.8, M1.9, M1.8, ...") until it runs out of tokens; on
`/inference/query`, a `repetition_penalty` of `1.1` stops it.

### Errors

Errors use the standard envelope; see the [Inference API errors](https://docs.archetypeai.app/api-reference/inference/about#errors).
The ones you are most likely to meet while migrating:

| Status | Message | What to do |
| - | - | - |
| `400` | `model '<model>' refused this request` | A field or input path isn't supported. Check the body against [Old Fields, New Fields](#old-fields-new-fields) |
| `404` | `model '<model>' is not available` | Use `newton-c-2.6` or `newton-fusion-1.0`, not the `Newton::` identifier |
| `413` | | The request body is too large. Shrink the clip or image |
| `503` | | No copy of the model is ready yet. Retry shortly |
| `504` | | The model didn't answer within about 60 seconds. Ask for a shorter answer, send a shorter clip, or answer without reasoning |

### No Memory Between Queries

Each query stands alone, as before. To ask a follow-up question, send the image or clip again
with the new question.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.