> ## Documentation Index
> Fetch the complete documentation index at: https://docs.archetypeai.app/llms.txt
> Use this file to discover all available pages before exploring further.
> ## Agent Instructions
> Start with /introduction/getting-started. Use the Direct Query API (POST /query) with the Newton Fusion model (text, image, and video reasoning) or the Newton Omega encoder (time-series embeddings). ATAI_API_ENDPOINT must include the version path: /v0.5 for most APIs, /v0.6 for the Fine-Tuning Service. Pages whose descriptions are marked (Archived) document the legacy Lens runtime — do not use them for new projects.
# v1.1.x
> These notes detail the new features, enhancements, and fixes included in each release of our platform.
## **API & Services**
* **All remaining list endpoints moved to cursor pagination**: The JOS registry lists and all
eight v0.5 batch lists (jobs, events, inputs, inputs/progress, outputs, progress and both
checkpoint lists) now use cursor pagination instead of the old `offset` and `total` response
values. In addition, the per-entity array keys on the batch lists have been replaced
by `data`. Each list keysets on `(created_at, id)` with new covering indexes, so deep pages
stay fast and a row changing mid-scan neither skips nor repeats.
* **The file list can be filtered by several MIME types at once**: `file_type` on `GET
/v0.5/files/metadata` (and the internal metadata endpoint) now accepts a comma-separated list
such as `image/png,image/jpeg` and returns files matching any entry. A single value behaves
exactly as before, blank entries are ignored, and more than 50 types is rejected with an HTTP
400 response.
* **The file list can be sorted**: `GET /v0.5/files/metadata` now lets you sort on creation
timestamp, file size, file status, and file ID in either ascending or descending order. It
uses sensible default sorting direction per field: newest and largest files first, statuses in
lifecycle order, names A to Z. The `created_at` timestamp is now exposed in the public
metadata as well.
* **Stored file types are now derived from the filename, not the uploading client**: Every file
upload path resolves `file_type` from the file extension, only falling back to the
client-supplied type for unknown extensions, so two identical files can no longer exist under
different types. `.json` and `.tsv` uploads are no longer rejected by the public upload
handlers, and TSV attribute extraction still runs. Existing rows are not backfilled.
* **The model list gained the filters a registry UI actually needs**: `GET /models` now
accepts `model_family`, `scope` (`platform` | `org` | `all`), and a
case-insensitive `search` over tag name, description, family and variant. All three AND with
each other and with the pre-existing `variant` filter; an unknown `model_family` is a 400
rather than an empty page, and LIKE metacharacters in `search` are escaped so user text
matches literally.
* **Newton Omega 1.5 added as a platform model.**
* **`GET /jobs` accepts the paused and interrupted statuses**: The v0.5 status parser
rejected `PAUSED` and `INTERRUPTED` with a 400 while v0.6 accepted them; both versions now
filter on the same status set.
## **Agents**
* **The blueprint and bundle lists answer what the agent manager needs**: `GET
/agents/blueprints?bundles_count=true` annotates each blueprint with the number of bundles
visible to the caller's organization that are pinned to it, so a client can offer only
blueprints that have agents. `GET /agents/bundles` accepts `sort` (with permitted values
`created_at`, `last_run_at`, and `name`) and `order`, keeping the existing last-run-recency
default.
## **Console & UI**
* **Redesigned the UX for the Model Registry:** The grid of cards has been replaced with a table
similar to the one used by the Batch Manager. Each row includes the parameters and context
length (metadata not previously disclosed anywhere in the UX). Also added is support for
search as well as model family and scope filters. The model details dialog has been redesigned
as well, with better labeling and hierarchical structure.
* **The Agent Manager can now be sorted**: A Sort menu sits between the search field and the
view toggle, offering last run, created, and name as sorting options, with a second activation
reversing the field and each direction named in the field's own words (Newest/Oldest for
dates, A–Z for names). The Build Status filter is gone since it was rare for an agent’s status
to be anything other than `Ready`, and the count was misleading due to the way it was
calculated.
* **The agents playground is embedded in the console**: A server-side reverse proxy serves the
agents playground app under `/playground/agents`, injecting the caller's credentials as
pass-through headers so the embedded app authenticates as the signed-in user. Cookies never
cross the proxy in either direction, and path and query are forwarded untouched, so mounting a
future playground app takes one list entry.
* **The agents pages moved to a clearer URL hierarchy**: The agent gallery moves
from `/dashboard/agents` to `/dashboard/blueprints`, joining the blueprint detail page already
there, and the agent manager takes over `/dashboard/agents`. Bundle and run detail pages now
nest under their parent, so a run reads as `/dashboard/agents/[id]/runs/[runId]`. Existing
bookmarks to the old paths will need updating.
* **The console side nav was consolidated**: The nav gains a stacked logo lockup and one flat
nav list, and the account dropdown becomes an always-visible profile block at the bottom — an
identity row (avatar, name, org · role, theme toggle) over a list of account rows, with Sign
Out as a normal row rather than a separate destructive button. A full-bleed separator divides
the block from the nav list above.
* **Copy buttons behave the same everywhere**: Copying was implemented three ways across the
Console, each with its own copied state, icons, delay and toast — or none, since seven places
gave no feedback at all on success. Every call site now uses the design
system's `copy-button` primitive, which owns copied state, feedback duration, toasting and
click isolation, so every copy shows an inline check and a toast.
* **Info tooltips render as a filled disc**: The console moved to design system 0.15.0 and
adopted the new `Tooltip.Trigger variant="info"`, which draws its own mono `i` inside a filled
20px disc with the arrow centered on all four sides. Call sites that passed their own icon are
unaffected, since children still take precedence.
## **Bug Fixes**
* **A required checkpoint port is configurable in the Create Batch Job modal**: The modal folded
a workflow's lone `checkpoint_ref` port into the `Model` select as a "Custom Checkpoint"
option, which is right when the port is optional and wrong when it is required
— `driving-behavior` needs a kNN classifier on the port *and* a summarizer through `model`, so
folding them made the workflow impossible to configure. A required port now renders as its own
control that can pick a promoted fine-tuned model from the registry, with a typed `ckp_` id as
the fallback.
* **The registry no longer offers dead links to fine-tuning jobs**: Models are org-scoped but
fine-tuning jobs are owner-scoped, so a member could see a model and click through to a job
they cannot open. Each lineage job is now looked up with the reader's own credentials when a
model dialog opens; a 404 hides the Job row, while any other failure leaves it in place.
* **A successful fine-tuning run no longer looks like a failure**: A run produces checkpoints,
not a model, so the `fine_tuned_model` Overview row was empty until someone promoted a
checkpoint — and it rendered as a bare em-dash, which reads as a job that failed. The row now
says "Not promoted yet" with an info tooltip explaining that a checkpoint has to be promoted
first, and the checkpoints table's Metrics column no longer collapses to 14px on the same
page.
## API & Services
* **Uploaded files now get a reliable integrity checksum on every upload path:** both the
streaming/proxy upload path and the direct-to-cloud upload path now persist a CRC32C checksum
on file metadata, closing a gap where the old MD5 checksum was only ever populated for the
legacy streaming path.
* **Job inputs can now be searched via the API:** `GET /jobs/{id}/inputs` accepts a
case-insensitive substring `search` query parameter against a file's name or file ID,
combinable with the existing port/status filters. This lets the console
search a job's inputs server-side instead of filtering an already-loaded page.
* **Jobs that get interrupted (e.g., by a pod eviction) can now recover automatically instead of
staying stuck forever:** the job orchestration service will retry an interrupted job up to a
configured retry budget, and will now actually fail a job once that budget is exhausted rather
than leaving it parked indefinitely.
* **The public Python client gained a runnable Anomaly Discovery example:** a new quick-start
script mirrors the existing Operational State Monitoring and Rare Event Detection examples —
it discovers the canonical Anomaly Discovery demo bundle, uploads sample sensor data, runs it,
and reads back per-window anomaly scores — giving developers a working reference for building
their own AD-based agents.
## Agents
* **Agent bundles now configure models per slot instead of a single shared model:** the old
`bundles.model` field has been replaced with `bundles.models`, a map from slot name to
registry tag, so a bundle can pin a different model to each of its slots. This is a breaking
change for any existing integration calling the public client's bundle-creation API with the
old `model=` argument, which is now `models=`.
* **Bundle creation now rejects broken model-slot references immediately instead of failing on
every subsequent run:** creating or updating a bundle with a malformed `${models...}`
override, or a slot name that can never actually be addressed, now returns a validation error
at creation time.
* **Agent artifact references now resolve correctly:** `agent://{id}` references to another
agent's output are resolved more robustly at bundle-creation time.
* **A new "driving behavior" analysis job type is now supported end to end:** a dedicated worker
and pipeline ingest trip data together with a fine-tuned checkpoint and produce behavior
summaries.
* **Long-running agent activities can now report that they're still alive**, preventing
legitimately long operations from being mistaken for a hang or failure.
## Console & UI
* **The console's navigation was rebuilt around a single persistent sidebar, replacing the old
top navigation bar:** every logged-in screen now shares one collapsible sidebar with a
remembered collapse state and tooltips when collapsed, and account-management pages (profile,
org admin, platform admin) get proper contextual navigation instead of unlabeled "back"
buttons.
* **Running certain agents now shows a progress bar during inference** instead of an unchanging
spinner.
* **The Agent Manager page now shows live run statuses** instead of only a static, last-known
state.
* **Anomaly Discovery and the Fusion models (Manual Generation Agent (MGA) and Task
Verification Agent (TVA)) now walk you through running the agent on your own data, not just
the built-in demo:** the "Try it out" flow for MGA and TVA leads with an install step and
then creating/running an agent bundle against your own video, and Anomaly Discovery gained
both a runnable example and a playground demo card.
* **File Manager and Batch Manager terminology was clarified:** "Lenses" has been replaced
with "agents" and "batch inference" is now "batch processing" in the File Manager UI, and a
Batch Manager job-detail field was relabeled "Workflow key" to match the corresponding
list-view column.
* **The Agents page dropped a redundant "Agent" label and card outline** for a cleaner look.
* **The agent configuration view now merges related configuration tables and shows reference
defaults, and the bundle list is now ordered by last run** so recently-used bundles surface
first.
* **Several list views — batch jobs, fine-tuning jobs, and files — were migrated onto the newer
design-system table component**, in some cases dropping an older grid view, for a more
consistent look and better performance.
* **The job-detail page now shows counts on its tab strips and groups its log feed by day**,
making long-running jobs easier to scan.
## Security & Authentication
* **API request validation is being hardened incrementally:** several job-orchestration
endpoints now reject requests containing fields they don't recognize (previously such fields
were silently ignored), and new declarative field validators plus organization-visibility
checks are rolling out across services — some enforced immediately, others running in
observe-only mode ahead of full enforcement.
## Bug Fixes
* **Fixed the agent manager losing your blueprint filter (and other list state) when you
navigated into an agent's detail page and back.**
* **Fixed the public Python client's test installation and Python version requirements.**
Installing the client's test extra (`pip install -e '.[test]'`) previously failed silently due
to an invalid `setup.py` keyword; that's fixed, the client's stated minimum Python version was
corrected to 3.10+ (matching what already worked in practice), and an incorrect upper bound
that would have blocked installation on Python 3.14 was removed.
* **Reliability fixes for agent memory usage and out-of-memory handling:** adjusted memory
limits, added an additional memory-pages reclaim pass, and improved out-of-memory handling to
reduce agent crashes under memory pressure.
## API & Services
* **Introduced the new Agents API:** The Agents API lets you run and manage agents. Agents are
bundles of resources that describe how to use Newton to perform a certain task. With an
agent configured to perform a needed task, all you need to do is provide the input file or
files and the agent will process your data and return the desired output.
* **Updated the Python SDK with agent support and added examples for the five canonical
agents:** Developers can now call these workflows directly using the Python public client
library.
* **Improved Fine-Tuning API support:** Fine-tuning jobs can now accept promoted fine-tuned
models as input and track fine-tuning lineage on promoted models.
## Agents
* Shipped our first five agents:
* **Anomaly Discovery**
* **Task Verification**
* **Manual Generation**
* **Operational State Monitoring**
* **Rare Event Detection**
* Use the Agents API to create new agents based on these agent blueprints, then run your
agents either using the API or the Developer Console.
## Console & UI
* **Added the Agent Gallery and Agent Manager dashboards:** View information about the
canonical agent blueprints using the Agent Gallery, and use the Agent Manager to manage and
run your agents from the Developer Console.
* **A Model Registry UI was added to the Console, including card views, promotion, and
deletion.** Users can browse registered models, promote a checkpoint to a model, and delete
org-scoped models directly from the interface.
* **Adopted a shared job-detail template across fine-tuning and agent runs:** Fine-tuning jobs
and agent runs now share a consistent job-detail layout, including a health report view and
a logs/events dialog on the runtime status page.
## Other
* **Added CSV file format support to job processing:** Jobs can now ingest CSV-formatted input
files.
## Bug Fixes
* **Fixed a grammar issue in the File Manager UI:** Corrected a run-on sentence in the File
Manager's upload help text.
* **Fixed the CLI error output:** The `atai-cli` now shows the full error chain instead of
only the outer context, making failures easier to diagnose.
* **Fixed agent Jobs Orchestration Service status handling:** The JOS client now correctly
handles `PAUSED` and `INTERRUPTED` job statuses.
## API & Services
* **Fine-tuning and serving accept a registry model tag alongside the legacy keyword**: Newton fine-tuning, Newton batch serving (`activity-detection`, `manual-generation`), and Omega fine-tuning plus machine-state jobs all run dual-mode — a job that sends no tag behaves exactly as before. Profile-delivered config takes precedence over the legacy `model_variant`/`presets` lookup at a single branch point, and serving a promoted Omega head now loads directly from the registry bundle instead of the legacy `fine_tune_checkpoint` port.
* **Create Job, Get Job, and List Jobs responses now return `workflow_key`**:These responses now carry the `workflow_key` property with the same value as `pipeline_key`, which is retained unchanged for backward compatibility. `workflow_key` is the preferred name going forward. The pipeline registry responses keep only `pipeline_key`. This completes changes introduced in v1.1.7.
* **Lens v2 supports live RTSP cameras again**: A new RTSP pipeline drives a live camera through the same register → buffer → submit path as the file-video pipeline, wire-compatible with the legacy v1 stream configuration (`rtsp_url`, `target_frame_rate_hz`, `target_image_size`). Each connection gets a dedicated decode thread handing frames over a drop-oldest queue, so a stalled consumer keeps the freshest frames rather than compounding latency, and camera drops reconnect with exponential backoff. Session destroy is now honored by all four reader loops within about a second, so a destroyed session stops reconnecting instead of burning its backoff budget.
* **JOS client supports the fine-tune API and KNN fine-tune jobs**: The client can submit Fine-Tuning API jobs and Omega KNN preparation jobs directly.
## Console & UI
* **Job detail pages converge on one shared layout**: Job detail pages across most of the UI have converged on a single design. Progress moved out of the tab strip to the top of the page as a "now" block, so the status banner no longer duplicates the progress bar for healthy running or cleanly succeeded jobs. Config moved from a rail card into its own full-width tab. Fine-tuning jobs have not yet adopted the new look.
* **Simplified and improved the profile and API key UI:** `/profile` now hosts profile details and API-key create/revoke together on one screen.
## Security & Authentication
* **Platform admins can deactivate and reactivate users from the console**: Two new superadmin endpoints and matching buttons on the org members table close the gap that previously forced a superadmin to fall back to CLI plus port-forward plus master key. Actions are logged against the specific user’s `user_id` rather than a generic `"admin"` actor, and a superadmin cannot deactivate their own account. Reactivation deliberately does not restore revoked API keys — that would defeat the offboarding cascade — so users mint fresh keys via SSO.
* **RTSP URL handling is hardened**: The URL scheme is now allowlisted to `rtsp`/`rtsps`. Loopback and link-local literal IPs are rejected to close off the cloud metadata endpoint and localhost side channels, while private LAN ranges stay allowed since VPN-reachable on-premises cameras are the primary real source. URL credentials are redacted from every log line and emitted event.
## Bug Fixes
* **Batch-creation errors now include details about how to resolve the problem**: About 20 strings across the create modal, file upload, and file pickers were rewritten so every message names the problem *and* the resolution, dropping developer jargon like raw JSON pointers and status codes.
* **Long agent names no longer hide the status badge**: The card header's flex wrapper is a grid item defaulting to `min-width:auto`, so it refused to shrink and pushed the badge past the card's clipped edge. Adding `min-w-0` lets the title truncate instead, and a `title` attribute makes the full name readable on hover.
* **Fine-tuning metrics reach the console again**: The metric produced during MSJ inference over the fine-tune job was not saved alongside the checkpoint, so it never propagated to the UI.
## API & Services
* **Fine-tuned checkpoints can now be promoted directly to fine-tuned models with base model tracking.** The base model tag is resolved and recorded at dispatch time, and both the checkpoint promotion and the resulting model responses now expose base model and tag information.
* **Fine-tuning jobs no longer report success after an internal failure.** Previously the worker could swallow an error and mark a failed fine-tuning job as successful, which is now corrected.
* **Batch inference now accepts a `workflow_key` parameter, deprecating the older `pipeline_type`/`pipeline_key` fields.** This simplifies how batch jobs specify which workflow to run.
## Console & UI
* **The Console sidebar was collapsed into an icon rail.** The refresh also adopted secondary call-to-action alert actions from the updated design system.
* **The Console service now validates its environment configuration on startup.** Misconfigured deployments fail fast with a clear error instead of surfacing confusing runtime issues later.
## Security & Authentication
* **A wave of Critical/High severity CVEs has been remediated across non-PDM build surfaces.** The fix updates vulnerable dependencies as part of an ongoing security hardening effort.
* **IAM now cascade-revokes a user's API keys when their account is deactivated.** A companion CLI command was added so operators can trigger the same revocation workflow manually.
## Bug Fixes
* **Role checks on checkpoint promotion and model deletion were tightened.** This closes a gap that could allow insufficiently privileged users to perform these actions.
* **Z-score computation for flatlined channels in omega\_1\_5 is now stable across different frameworks and hardware.** This fixes inconsistent results that previously varied depending on the runtime environment.
## API & Services
* **Lens Runner v2 is now enabled by default.**
* **GPQ now distinguishes cancelled queries from failed ones and fixes its default `max_frames`
setting.**
## Console UI
* **The Console UI underwent general normalization to bring a more cohesive feel to the
interface.**
* **Batch inference jobs in the Batch Manager can now take a checkpoint as an input.** This is
shown using distinct UI treatment on the job's input files tab.
## IAM & Authentication
* **Organization-admin and platform-admin capabilities have been added to IAM, gated by new
JWT-scoped middleware.** A new route group requires a JWT with `sub_type=user` and
`role=admin`, unblocking Console pages for full organization member and settings management
as well as a platform-wide super-admin grant and bulk member creation.
* **The `owner_user_id` predicate is now enforced on files, jobs, lenses, and sessions,
scoping resource visibility to the requesting user.** This continues our work on per-user
resource scoping and applies the same visibility predicate consistently across the data,
JOS, and lens services.
## Fine-Tuning Service
While most Archetype platform endpoints are under API version 0.5, the Fine-Tuning Service
API is mounted under API version 0.6.
* **A new v0.6 fine-tuning API has been introduced to JOS, providing a user-facing interface
for creating and managing model training jobs.** The API is mounted at `/v0.6/fine_tuning`
and supports Newton and Omega model discrimination, cursor-paginated job listings,
checkpoint endpoints, and per-model hyperparameter schemas. It maps high-level fine-tuning
requests onto internal JOS training jobs and mirrors the OpenAI fine-tuning job object shape
in its responses.
* **The dedicated fine-tuning runner node has been retired and replaced by the general-purpose
training worker infrastructure.** This consolidates fine-tuning execution through the
standard worker path and removes the now-obsolete standalone runner component from the
platform manifest. *The Fine-Tuning Node API has been deprecated and is no longer available
for use.*
* **A fine-tuning dashboard has been added to the console for creating, managing, and
monitoring training jobs.** The page is modeled after the existing Batch Manager and
includes a job list/grid view and a create-job modal scoped to training pipelines. Batch
Manager pipeline listings have been updated to filter out training pipelines so the two
sections remain distinct.
## Job Orchestration Service
* **Pause and Resume endpoints have been added to JOS along with new `Paused` and
`Interrupted` job states.** Cancel has been made a terminal transition (the `Cancelled`
state is now final), and Retry now returns a 501 directing callers to Pause instead. The new
`Interrupted` state covers resumable worker failures separately from the non-resumable
`Failed` state, and resume is permitted from `Interrupted`.
* **Retry endpoint has been disabled.** The [Retry Job](/api-reference/batch/jobs/retry-job)
endpoint has been deactivated for the time being. Using it results in a `501 Not
Implemented` error. This endpoint will return with a future release.
* **Completed Kubernetes Jobs are now torn down immediately upon completion to reclaim node
disk space.** Previously only failed jobs triggered teardown, leaving model weight caches
(\~16 GB per pod) on GPU nodes for up to an hour, which caused intermittent "Insufficient
cache space" failures for subsequent fine-tuning jobs on single-node GPU pools.
* **Upfront validation for JSONL and CSV job inputs has been added, checking files before job
creation rather than at runtime.** File extension is now checked against the `text/plain`
MIME type, and input selection validates types at the point of selection.
* **A job outcome field has been added to JOS, and all jobs that finish execution are now
marked `Completed` regardless of whether individual inputs had processing errors.** This
separates jobs that completed their run from jobs that failed to run at all, giving a more
accurate picture of job health.
* **Processed bytes are now included in the progress summary returned with each job
response.** This data powers the byte-based progress display in the Batch Manager job
listing page.
* **Status information for job outputs has been added to JOS job responses.**
## Console UI
* **The Batch Manager has been significantly redesigned, introducing outcome modeling, a
refreshed sidebar, a new job-details page, and byte-based progress tracking.** Banner-style
callouts for job outcomes (succeeded, partially processed, failed, cancelled), and
byte-level progress from the server are now surfaced in the job list view.
* **Pause and Resume buttons have been added to the Batch Manager UI, allowing users to
suspend and continue in-progress jobs.** The Cancel CTA has been restored alongside the new
controls so all three lifecycle actions are available from the same interface.
* **The "Task Type" label in the Batch Manager UI has been renamed to "Workflow Key" for
clarity.**
* **Byte-level progress information on the Batch Manager job listing page has been fixed to
display accurate values.**
## IAM & Authentication
* **Bulk-revocation of all active API keys for one or more orgs has been added to the IAM
service and CLI.** A new master-key-protected endpoint (`POST
/v1/iam/orgs/{org_id}/keys/revoke-all`) performs a single atomic update and emits per-key
and summary audit events; the `atai iam org revoke-keys` CLI command defaults to a dry-run
preview and requires `--execute` to apply changes.
* **Structured audit events have been added across the full OIDC login flow in the console.**
Every meaningful state transition — authorize start, callback received, state validation
failure, KC/IAM exchange failures, and the existing success/failure events — now emits a
structured event with a shared `flow_id` UUID that correlates all steps for a single login
attempt. No raw secrets or token values are logged.
* **The `owner_user_id` column has been added to files, lenses, sessions, and JOS jobs to
track resource ownership at the user level.**
## API & Services
* **The Jobs Orchestration Service (JOS) now supports chunked file outputs, exposing a single
logical output file per input regardless of how many segments a worker produces.** Worker pods
tag each segment result with a `chunk_of` field containing the chunk index, parent filename,
and a final-chunk flag. Output URIs are rewritten to the authenticated `/files/download`
endpoint, and files carry a `job_output.status` attribute (`partial` while in progress,
`complete` when done) so users can track job state from the file library.
* **JOS now detects and reports stuck jobs through a new background sweep and an OTel
`jobs_stuck` gauge.** A job is now correctly marked `RUNNING` only when its first pod becomes
`Ready`, so jobs queued without available GPU remain in `ADMITTED` rather than prematurely
appearing active to users. The sweep detects two conditions — a job stuck in `ADMITTED` for
over an hour and a `RUNNING` job with no emitted event or progress for over an hour — and
exposes both counts via the new metric.
* **JOS workers now exit gracefully on job cancellation by catching `SIGTERM`, checkpointing the
current segment, and terminating cleanly.** The pod termination grace period has been raised
from 30 seconds to 2 minutes by default to give workers sufficient time to complete a
checkpoint before Kubernetes forcibly terminates the pod. The grace period is configurable per
deployment.
* **A manual-generation JOS pipeline has been added for producing structured procedural
documentation from video.** The pipeline performs audio-grounded batch inference and is
available as a new task type configured via a JOS seed deployed to the environment.
## Security & Authentication
* **API keys can now be created with an optional TTL, and expired keys are automatically marked
in the database on first use after expiry.** A new `--ttl-days` flag on the `atai iam key
create` and `atai iam user key create` CLI commands computes an `expires_at` timestamp at
creation time and passes it through to the IAM service. When an expired key is rejected, its
status transitions from `active` to `expired`. No schema migration is required as the
`expires_at` column already existed on the `api_keys` table.
## API & Services
* **Chunked files can now be downloaded via the existing `/files/download` endpoint, with
content streamed directly from S3 in chunk order:** The download path detects whether a file
is chunked and assembles the response as a streaming byte sequence across all stored S3
objects, with no need for a separate endpoint or client-side logic.
* **A new `SegmentedInferenceSession` abstraction has been added to the JOS worker client,
enabling automatic checkpointing, progress reporting, and resilient resume for batch
inference jobs:** Workers now iterate over "segments" (subsets of input files) rather than
whole files, and the session handles checkpointing between segments, emitting
file-started/completed events, and resuming from the last checkpoint on restart. Segment
size is configurable with defaults of 50 MB of input or 10 minutes, whichever comes first.
The Nano inference worker and the inference phase of the machine state job have been updated
to use this abstraction.
* **Stale file uploads stuck in the `UPLOADING` state are now automatically cleaned up:** A
configurable scheduler (default: every 6 hours) aborts any upload older than a threshold
(default: 24 hours) using the same logic as the `/abort` endpoint.
* **A new `DatasetList` port mode and a `PortMode::Checkpoints` type have been added to JOS to
support dataset listing and checkpoint upload workflows:** These additions enable workers to
declare input/output ports that carry dataset lists and model checkpoints respectively,
broadening the range of job types that can be expressed natively in JOS config. Checkpoint
files are now staged under a non-tmpfs folder to avoid memory pressure.
## Inference & Model Updates
* **The C2.6 model is now available with a new vLLM inference backend that delivers improved
throughput over previous transformer-based inference:** Multi-image support has been added to
Direct Query and the Nano engine, and GPU-specific YAML configurations allow fine-grained
tuning per model and hardware combination.
* **GPU memory budget calibration for C2.5 and C2.6 has been improved with a more accurate
adaptive probe sweep across batch size and shape dimensions:** The calibration now accounts
for padding-aware batch peaks, shrinks the memory budget incrementally after each OOM, and
warms up the engine at max probe shapes before calibrating. This reduces spurious OOM events
during production batch inference workloads.
* **UFM training has been extended with variable-K Stage 1+2 resolution,
`pad_mode=interpolate` retraining, Stage 3 sweep plumbing, and several stability fixes:** A
deterministic eval fix and a PEP 479 `StopIteration` correction were also merged, along with
pinning `flash-linear-attention` to avoid a \~3× training slowdown introduced by a dependency
update. The UFM builder and model directories were refactored to align with the factory
pattern introduced in the updated Nano engine.
## Observability & Telemetry
* **OTel SQL tracing has been extended to cover database transactions via a new
`TracedTransaction` wrapper, and rolled out to the Lens API and Data services:** This
completes full OTel tracing coverage for `data_service` and `lens_api_service`;
`iam_service` and `jos_service` will follow in subsequent releases.
* **New Python `OtelMetricsBase` and `OtelWorkerMetricsBase` base classes have been
introduced, and the first batch of active platform services has been migrated from
Prometheus to OpenTelemetry metrics:** The new base classes use the global `MeterProvider`
directly, dropping the per-service `CollectorRegistry` and `update()` chain used by the
legacy Prometheus classes. Services migrated in this release include `api_service`,
`lens_service`, and `gpq_service`; the previous `service_info` gauge is replaced by OTel
resource attributes.
* **Prometheus business metrics in `data_service`, `iam_service`, `jos_service`, `lens_api`,
`health_service`, and `registry_service` have been migrated to the OTel push model:** This
brings the Python service telemetry pipeline in line with the Rust services and enables
unified metric collection through the OTLP exporter. Per-request `org_id` propagation has
also been added to OTLP logs, and per-signal OTLP endpoint overrides are now supported for
routing traces, metrics, and logs to different collectors.
* **New Grafana dashboard panels have been added for JOS job counts, node pool sizes, total
file counts and bytes, and the sync has been improved to include fixes for overcounting:**
The panels provide visibility into per-org file storage volume and JOS queue depth and were
synchronized with the cloud environment. A separate fix corrects an over-counting bug in the
file count and bytes panels introduced during the initial rollout.
## Console & UI
* **Filename search has been added to the files dashboard, backed by new database indexes for
performant queries:** Users can now filter the file list by partial filename match directly
in the UI, and the underlying data service database has been updated with the necessary
indexes to keep query latency low even for large file tables.
* **The Console login page, homepage, and top navigation have been migrated to the Archetype
design system:** Light/dark mode persistence is preserved.
* **The Console dashboard and workbench pages have been migrated to the design system:**
Dashboard pages now use design system primitives, and the workbench has been updated while
keeping all existing functionality intact.
## Bug Fixes
* **vLLM batch job result ordering and crash recovery have been fixed to ensure JSONL output
is always in input order and that fatal engine failures are handled cleanly:** When async
vLLM workers complete out of order, results are now staged by submit position and flushed in
contiguous runs, keeping output aligned with the original input. Per-record failure messages
are sanitized to "inference error" so internal exception details never reach
user-downloadable files. When the vLLM engine encounters a fatal error (such as CUDA
out-of-memory), the worker now emits an `inference.engine_dead` event, cancels in-flight
work, and exits cleanly so Kubernetes restarts the pod and JOS resumes from the last
checkpoint.
* **A TOCTOU race condition and stale counter bug in chunked file writes have been fixed, and
chunk data and S3 objects are now properly cleaned up on file deletion:** Concurrent
appenders previously could race on `chunk_index` assignment; the fix ensures all writes
happen within a single locked transaction. Deleting a chunked file now removes all
associated `file_chunks` rows and their corresponding S3 objects, preventing orphaned
storage.
## API & Services
* **`file_type` filter added to the `list-files` endpoint:** A new query parameter allows callers to filter the file listing by MIME type or file type category, making it easier to retrieve only the files relevant to a specific workflow.
* **OpenAPI schema for `/query` expanded with full request/response documentation:** The OpenAPI spec for the `/query` endpoint now includes complete request and response schemas, improving SDK generation quality and developer documentation.
* **`ATAI_CA_BUNDLE_PATH` environment variable renamed to `ATAI_CA_BUNDLE`:** The legacy environment variable name was simplified for consistency. Deployments using the old name will need to update their configuration.
* **Per-org file count and byte total exposed as Prometheus metrics (PLDEV-784):** Two new Prometheus gauges track total file count and aggregate storage usage per organisation, enabling capacity planning and billing dashboards.
* **The `jos seed apply` command now bundles seed YAMLs (PLDEV-909):** The `jos seed apply` CLI command was updated to ship seed YAML files directly with the binary, so applying default data seeds no longer requires a separate file distribution step.
* **Example automatic migration scripts added for 1.0.9 → 1.1.1 and 1.1.1 → 1.1.2:** Reference migration examples were committed to the repository so operators have a concrete starting point for upgrading existing deployments across these version boundaries.
## Observability & Telemetry
* **Full OpenTelemetry OTLP distributed tracing stack added to all Rust services:** The `atai_telemetry` crate now wires a complete three-signal OTel pipeline — distributed traces, log records with trace/span correlation, and push metrics — all exported via OTLP/gRPC when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. HTTP spans follow OTel semantic conventions via `axum-tracing-opentelemetry`, with W3C `traceparent` extraction and propagation across all four Rust HTTP services. When the endpoint is unset, the crate behaves exactly as before with zero OTel overhead.
* **OTLP log bridge and trace improvements added to `console_2_service`:** This PR wires the OTLP log bridge into `console_2_service` and improves trace context handling, including always-on W3C trace ID generation and `X-Request-Id` reflection in responses. Static asset requests are excluded from tracing noise, and root HTTP span naming now uses `METHOD /route/[param]` with SvelteKit route IDs.
* **`init_tracer()` added to `atai_py` and wired into `initialize_service_logging`:** Python services can now initialize OpenTelemetry tracing with a single call, automatically reading `OTEL_EXPORTER_OTLP_ENDPOINT` and configuring the tracer pipeline (PLDEV-801).
* **`PLATFORM_VERSION` is now propagated through logs and metrics:** The platform version string is attached as a resource attribute on all telemetry signals, making it easier to correlate observability data with specific release versions.
* **Legacy `x-trace-id` header dropped in favor of the OTel `traceparent` standard:** The proprietary `x-trace-id` request/response header has been removed and all tracing correlation now relies on the W3C `traceparent` header (PLDEV-797). Any tooling or dashboards relying on `x-trace-id` will need to be updated.
* **`atai_telemetry_reqwest` crate adds OTel HTTP client spans via `reqwest-middleware`:** Outgoing HTTP calls made through `reqwest` now produce properly attributed OTel client spans (PLDEV-798), enabling end-to-end trace stitching for calls leaving Rust services.
## Security & Authentication
* **JWT exchange endpoint and JWKS added to the IAM service:** A new `POST /v1/iam/exchange` endpoint accepts an API key and returns a signed RS256 JWT containing Archetype claims (subject, org ID, role, auth method), laying the foundation for moving from per-request API key validation to JWT-based auth. A companion `GET /v1/iam/.well-known/jwks.json` endpoint serves the RSA public key so downstream services can validate tokens locally. The existing `/v1/iam/authenticate` endpoint is unchanged for backward compatibility.
* **`UPLOADING` status is now exposed in the public file API (PLDEV-833):** Previously the `UPLOADING` state was hidden from the public-facing file status endpoint; it is now surfaced so clients can accurately track in-progress uploads.
## Data Integrity & Uploads
* **End-to-end server-driven checksum verification added to the direct upload flow (PLDEV-663):** The server now selects the checksum algorithm at upload initiation and returns it in the `InitiateUploadResponse`; clients can optionally compute and submit a whole-file CRC32C in `CompleteUploadRequest`. If provided and mismatched, the file is marked `CORRUPT` and HTTP 422 is returned; if absent, the upload succeeds without verification, preserving backward compatibility. S3 `CreateMultipartUpload` is called with `ChecksumType: FullObject` so a single whole-object CRC32C is stored and retrievable via `HeadObject`.
## Inference & Model Features
* **vLLM engine support added along with c26 improvements:** The vLLM inference engine backend was integrated, including model weight caching and related configuration changes for the c26 hardware generation.
* **Stage 3 task classification expanded with MoteStrain and PAMAP2 datasets for UFM (fixed and variable):** The UFM task classification pipeline now supports MoteStrain and PAMAP2 benchmarks across both fixed and variable input configurations.
* **Jobs now fail explicitly if any input fails (PLDEV-940):** Previously a job could silently succeed even if one or more of its inputs errored; the job runner now propagates input failures and marks the overall job as failed.
### Console & UI
* **Autocomplete label suggestions added to the n-shot file picker (PLDEV-715):** As users enter class labels across n-shot files, a session-scoped vocabulary is built up and previously used labels are surfaced as autocomplete suggestions on subsequent inputs. The dropdown supports keyboard navigation (Arrow Up/Down, Enter, Escape), mouse selection, and auto-scrolls the active suggestion into view.
* **Error tooltip in the console stays visible longer and supports text copying:** The tooltip that appears on API or service errors now remains on screen long enough to read and can have its content copied, improving the debugging experience for users (PLDEV-786).
* **Content-aware column widths applied to batch manifest tables:** Columns in batch manifest views now size themselves based on their content rather than using fixed widths, improving readability for a wide range of payload shapes.
* **Progress chart tooltip is now flipped when near the right edge (PLDEV-889):** The tooltip on progress charts was being clipped when the cursor was near the right boundary; it now flips to the left to stay fully visible.
* **"Pipeline" label renamed to "Task Type" throughout the console (PLDEV-905):** The UI label used to describe processing pipelines has been standardized to "Task Type" for consistency with the rest of the product terminology.
* **MSJ broken image display fixed:** A regression that caused broken image previews in the multi-sensor join UI was resolved.
* **Wrong progress counter in MSJ fixed (RES-272):** A display bug that caused incorrect
progress percentages to appear in the MSJ task view was corrected.
## New Features & Improvements
* **Added direct-to-cloud file upload support to the Rust and Python SDKs (PLDEV-535,
PLDEV-16):** Both the Rust and Python SDKs now support uploading files directly to cloud
storage via presigned S3 URLs, bypassing the data service proxy. The Rust SDK introduces a new
builder API (`UploadBuilder`) as the default path, with the proxy path still available via
`.using_proxy()`; the Python SDK makes the direct path opt-in via `use_proxy=False`. Both
implementations support concurrent multipart uploads with configurable worker counts, per-part
retries with exponential backoff, progress callbacks, and cancellation. This enables uploads
well beyond the proxy's previous 500 MB size limit.
* **Added resumable upload support to the Rust and Python SDKs (PLDEV-551, PLDEV-778):** Clients
that fail mid-upload can now resume from where they left off without re-uploading
already-completed parts. A new server-side checkpoint endpoint stores completed part tokens,
and the initiate call accepts a `resume_if_started` flag to reuse an in-progress upload for
the same file. On the Rust SDK, resuming is enabled via `.with_resume(true)` on the upload
builder, with checkpointing on by default; on the Python SDK, resuming is enabled by passing
`allow_resume=True` to `upload()`, with checkpointing also on by default. If a progress
callback is set, it will be called once with the already-uploaded byte count when a resume
occurs.
* **Increased the maximum upload file size to 250 GB:** The platform-wide maximum file size for
uploads has been raised to 250 GB to accommodate large dataset and model artifact transfers.
* **Added a session validation step to the Workbench (PLDEV-195, PLDEV-600):** The Workbench now
sends an explicit `session.validate` event at the start of every lens session before entering
the active streaming state. If validation fails, users receive a clear error notification even
when the session log panel is collapsed, and error log entries are highlighted in red for
quick visual identification. This addresses a recurring issue where heartbeat timeouts would
cause sessions to continue in a degraded state without surfacing clear feedback to users.
* **Added a read-only selected file name field to the Workbench lens tray (PLDEV-609):** The
lens tray settings panel for the Activity Monitor and Machine State lenses now displays the
name of the currently selected input file below the model version field. When no file has been
chosen, the field shows "Not selected."
* **Limited CSV table and graph rendering to 10,000 rows in the Workbench to prevent freezes on
large datasets (PLDEV-186):** When an uploaded CSV file exceeds 10,000 rows, the Workbench now
truncates the preview display and shows a banner informing users of the total row count and
directing them to the API for full data access. CSV data is now truncated at the line-split
stage before full parsing occurs, preventing memory pressure from very large files.
* **Improved Workbench output panel autoscroll behavior (PLDEV-350):** The Workbench output
panel now auto-scrolls to the latest response by default but pauses when the user manually
scrolls up to review earlier results. A **NEWEST** button appears when the user has scrolled
away from the bottom, and clicking it jumps back to the latest output and resumes
auto-scrolling. Autoscroll state is reset at the start of each new session.
* **Added drag-and-drop bulk file upload to the File Manager (ATAI-2938):** Users can now upload
files to the File Manager by dragging and dropping them directly onto the file list page, or
by using a new modal-based upload dialog triggered from the "Add files" button. The upload
dialog shows per-file progress, supports individual file cancellation, and retains failed
upload placeholders in the list so users can see which uploads did not complete.
* **Added the Batch Manager to the Console with live job creation, listing, and detail pages
(PLDEV-575, PLDEV-576, PLDEV-604, PLDEV-689, PLDEV-695):** A new Batch Manager page is
available in the Console, allowing users to create and monitor batch jobs submitted to the Job
Orchestration Service, and to download job artifacts. See the [Batch Manager
documentation](https://docs.archetypeai.app/developer-console/batch-manager) for details.
* **Added Python JOS clients for API access and job container use (PLDEV-348):** Two new Python
clients for the Job Orchestration Service have been added. `JosApiClient` covers all 20 REST
API endpoints (jobs, components, pipelines) with an async-first design and structured error
handling. `JosWorkerClient` is for use inside JOS-managed pods and provides
`InputPort`/`OutputPort` abstractions for reading JSONL manifests from S3, uploading outputs,
reporting progress via Redis, and saving and restoring checkpoints.
* **Synced CSV `window_size` and `step_size` from the Workbench lens tray to the input stream
config (PLDEV-629):** When a user updates the `window_size` or `step_size` fields in the CSV
lens tray, the values are now propagated to the underlying input stream config so that
inference window boundaries correctly reflect the configured parameters.
* **Added structured error responses to the data service (PLDEV-591):** The data service now
returns structured, machine-readable error response bodies across its endpoints, replacing
unstructured text errors.
## Bug Fixes
* **Fixed lens worker crash loops from missing `model_parameters` and stale Redis events
(PLDEV-610):** A cascading failure was identified where missing `model_parameters` in a lens
config caused a `KeyError` crash on worker nodes, and stale Redis queue events from crashed
sessions caused nodes to re-enter crash loops on restart. Five targeted fixes prevent the
`KeyError`, drain stale Redis events on node restart, skip unknown-session events, and
garbage-collect stuck sessions.
* **Fixed narrator memory not being initialized for direct `model.query` calls without a video
stream (PLDEV-632):** When `model.query` was called without a preceding `stream.start` event —
for example, for non-video file inputs — the narrator's internal memory and buffer fields were
uninitialized, causing failures.
* **Fixed deleted files returning stale data from the Files API (PLDEV-614, PLDEV-605):** The
`GET /files/metadata/{id}` endpoint now returns 404 Not Found on deleted files instead of the
old record with an "unknown" status, and the download endpoint also returns 404 instead of 400
on deleted files. Re-uploading a file to a previously deleted file key is now permitted.
* **Fixed multipart upload completion not accepting unsorted parts (PLDEV-670):** The
`/files/uploads/{id}/complete` endpoint no longer requires the uploaded parts list to be
submitted in sorted order.
* **Fixed the `window_size` and `step_size` values not being returned by the console-2-service
(PLDEV-600):** The backend was not including `window_size` and `step_size` in its responses,
causing the lens tray to display stale or missing values after a session was initialized.
* **Fixed the console CSV config tooltip width and wrapping:** The tooltip for CSV configuration
settings in the console was overflowing its container; width and text-wrapping constraints
have been applied.
* **Fixed the console-2-service to correctly apply schema default values for config templates:**
The console frontend now correctly applies schema-defined default values when rendering config
template fields, rather than leaving them empty.
## New Features & Improvements
* **Added the Fine-Tuning Node to perform fine tuning jobs on dedicated GPUs:** The new
Fine-Tuning Node offers an API to create, manage, and monitor fine-tuning jobs for your
organization. Each Fine-Tuning Node runs a fine-tuning job on its assigned GPU, acting like a
worker that trains a model using the provided dataset and configuration. This produces a
fine-tuned model and training metrics.
* **Added the Batch Manager to the Console with live job creation and listing (PLDEV-575,
PLDEV-576):** A new Batch Manager page is now available in the Console (behind the
`CONSOLE_FLAG_SHOW_BATCH_MANAGER` feature flag), allowing users to create and monitor batch
jobs submitted to the Job Orchestration Service. The Batch Manager supports creating jobs with
file input selection and YAML/JSON config validation, displays submitted jobs with status
badges and relative timestamps, and fetches job and pipeline data server-side.
* **Added a session validation step to the Workbench (PLDEV-195, PLDEV-600):** The Workbench now
runs an explicit `session.validate` event at the start of every lens session before entering
the active streaming state. If validation fails, users receive a clear error notification even
when the session log panel is collapsed, and error log entries are highlighted in red for
quick identification.
* **Added a read-only selected file name field to the Workbench lens tray (PLDEV-609):** The
lens tray settings panel for the Activity Monitor and Machine State lenses now displays the
name of the currently selected input file below the model version field. When no file has been
chosen, the field shows "Not selected."
* **Limited CSV table and graph rendering to 10,000 rows in the Workbench to prevent freezes on
large datasets (PLDEV-186):** When an uploaded CSV file exceeds 10,000 rows, the Workbench now
truncates the preview display and shows a banner informing users of the total row count and
directing them to the API for full data access.
* **Improved Workbench output panel autoscroll behavior (PLDEV-350):** The Workbench output
panel now auto-scrolls to the latest response by default but pauses scrolling when the user
manually scrolls up to review earlier results. A "NEWEST" button appears when the user has
scrolled away from the bottom, allowing them to jump back to the latest output and resume
auto-scrolling. Autoscroll state is reset at the start of each new session.
* **Added drag-and-drop bulk file upload to the File Manager (ATAI-2938):** Users can now upload
files to the File Manager by dragging and dropping them directly onto the file list page or by
using the new modal-based upload dialog. The upload dialog supports multiple concurrent file
uploads, individual cancellation, progress tracking per file, and persists failed upload
placeholders in the file list so users can see which uploads did not complete.
* **Implemented the Machine State Job (RES-226):** A new batch job for machine state
classification has been added, built on the fine-tuned Omega 1.3 model. The job can run on
both CPU and GPU, processes sensor CSV data, and supports n-shot input files for healthy and
faulty reference examples.
* **Synced CSV `window_size` and `step_size` from the Workbench lens tray to the input stream
config (PLDEV-629):** When a user updates the `window_size` or `step_size` fields in the CSV
lens tray configuration, the values are now propagated to the underlying input stream config
so that the inference window boundaries correctly reflect the configured parameters.
## Bug Fixes
* **Fixed narrator memory not being initialized for direct `model.query` calls without a video
stream:** When `model.query` was called without a preceding `stream.start` event (e.g., with
non-video file inputs), the narrator's memory and buffer fields were uninitialized, causing
the lens service to fail.
* **Fixed deleted files returning stale data from the Files API (PLDEV-614, PLDEV-605):** The
`GET /files/metadata/{id}` endpoint now returns a `404 Not Found` response when called on a
deleted file, rather than returning the old file record with an "unknown" status. The `GET
/files/download/{id}` endpoint similarly now returns `404` on deleted files instead of a `400`
error. Additionally, the Files API now allows re-uploading a new file over a previously
deleted file's key.
* **Fixed the console CSV config tooltip width and wrapping:** The tooltip for CSV configuration
settings in the console was overflowing its container; width and text-wrapping constraints
have been applied.