v1.1.x

​

API & Services

• Uploaded files now get a reliable integrity checksum on every upload path: both the streaming/proxy upload path and the direct-to-cloud upload path now persist a CRC32C checksum on file metadata, closing a gap where the old MD5 checksum was only ever populated for the legacy streaming path.
• Job inputs can now be searched via the API: GET /jobs/{id}/inputs accepts a case-insensitive substring search query parameter against a file’s name or file ID, combinable with the existing port/status filters. This lets the console search a job’s inputs server-side instead of filtering an already-loaded page.
• Jobs that get interrupted (e.g., by a pod eviction) can now recover automatically instead of staying stuck forever: the job orchestration service will retry an interrupted job up to a configured retry budget, and will now actually fail a job once that budget is exhausted rather than leaving it parked indefinitely.
• The public Python client gained a runnable Anomaly Discovery example: a new quick-start script mirrors the existing Operational State Monitoring and Rare Event Detection examples — it discovers the canonical Anomaly Discovery demo bundle, uploads sample sensor data, runs it, and reads back per-window anomaly scores — giving developers a working reference for building their own AD-based agents.

​

Agents

• Agent bundles now configure models per slot instead of a single shared model: the old bundles.model field has been replaced with bundles.models, a map from slot name to registry tag, so a bundle can pin a different model to each of its slots. This is a breaking change for any existing integration calling the public client’s bundle-creation API with the old model= argument, which is now models=.
• Bundle creation now rejects broken model-slot references immediately instead of failing on every subsequent run: creating or updating a bundle with a malformed ${models...} override, or a slot name that can never actually be addressed, now returns a validation error at creation time.
• Agent artifact references now resolve correctly: agent://{id} references to another agent’s output are resolved more robustly at bundle-creation time.
• A new “driving behavior” analysis job type is now supported end to end: a dedicated worker and pipeline ingest trip data together with a fine-tuned checkpoint and produce behavior summaries.
• Long-running agent activities can now report that they’re still alive, preventing legitimately long operations from being mistaken for a hang or failure.

​

Console & UI

• The console’s navigation was rebuilt around a single persistent sidebar, replacing the old top navigation bar: every logged-in screen now shares one collapsible sidebar with a remembered collapse state and tooltips when collapsed, and account-management pages (profile, org admin, platform admin) get proper contextual navigation instead of unlabeled “back” buttons.
• Running certain agents now shows a progress bar during inference instead of an unchanging spinner.
• The Agent Manager page now shows live run statuses instead of only a static, last-known state.
• Anomaly Discovery and the Fusion models (Manual Generation Agent (MGA) and Task Verification Agent (TVA)) now walk you through running the agent on your own data, not just the built-in demo: the “Try it out” flow for MGA and TVA leads with an install step and then creating/running an agent bundle against your own video, and Anomaly Discovery gained both a runnable example and a playground demo card.
• File Manager and Batch Manager terminology was clarified: “Lenses” has been replaced with “agents” and “batch inference” is now “batch processing” in the File Manager UI, and a Batch Manager job-detail field was relabeled “Workflow key” to match the corresponding list-view column.
• The Agents page dropped a redundant “Agent” label and card outline for a cleaner look.
• The agent configuration view now merges related configuration tables and shows reference defaults, and the bundle list is now ordered by last run so recently-used bundles surface first.
• Several list views — batch jobs, fine-tuning jobs, and files — were migrated onto the newer design-system table component, in some cases dropping an older grid view, for a more consistent look and better performance.
• The job-detail page now shows counts on its tab strips and groups its log feed by day, making long-running jobs easier to scan.

​

Security & Authentication

• API request validation is being hardened incrementally: several job-orchestration endpoints now reject requests containing fields they don’t recognize (previously such fields were silently ignored), and new declarative field validators plus organization-visibility checks are rolling out across services — some enforced immediately, others running in observe-only mode ahead of full enforcement.

​

Bug Fixes

• Fixed the agent manager losing your blueprint filter (and other list state) when you navigated into an agent’s detail page and back.
• Fixed the public Python client’s test installation and Python version requirements. Installing the client’s test extra (pip install -e '.[test]') previously failed silently due to an invalid setup.py keyword; that’s fixed, the client’s stated minimum Python version was corrected to 3.10+ (matching what already worked in practice), and an incorrect upper bound that would have blocked installation on Python 3.14 was removed.
• Reliability fixes for agent memory usage and out-of-memory handling: adjusted memory limits, added an additional memory-pages reclaim pass, and improved out-of-memory handling to reduce agent crashes under memory pressure.

​

API & Services

• Introduced the new Agents API: The Agents API lets you run and manage agents. Agents are bundles of resources that describe how to use Newton to perform a certain task. With an agent configured to perform a needed task, all you need to do is provide the input file or files and the agent will process your data and return the desired output.
• Updated the Python SDK with agent support and added examples for the five canonical agents: Developers can now call these workflows directly using the Python public client library.
• Improved Fine-Tuning API support: Fine-tuning jobs can now accept promoted fine-tuned models as input and track fine-tuning lineage on promoted models.

​

Agents

• Shipped our first five agents:
  • Anomaly Discovery
  • Task Verification
  • Manual Generation
  • Operational State Monitoring
  • Rare Event Detection
• Use the Agents API to create new agents based on these agent blueprints, then run your agents either using the API or the Developer Console.

​

Console & UI

• Added the Agent Gallery and Agent Manager dashboards: View information about the canonical agent blueprints using the Agent Gallery, and use the Agent Manager to manage and run your agents from the Developer Console.
• A Model Registry UI was added to the Console, including card views, promotion, and deletion. Users can browse registered models, promote a checkpoint to a model, and delete org-scoped models directly from the interface.
• Adopted a shared job-detail template across fine-tuning and agent runs: Fine-tuning jobs and agent runs now share a consistent job-detail layout, including a health report view and a logs/events dialog on the runtime status page.

​

Other

• Added CSV file format support to job processing: Jobs can now ingest CSV-formatted input files.

​

Bug Fixes

• Fixed a grammar issue in the File Manager UI: Corrected a run-on sentence in the File Manager’s upload help text.
• Fixed the CLI error output: The atai-cli now shows the full error chain instead of only the outer context, making failures easier to diagnose.
• Fixed agent Jobs Orchestration Service status handling: The JOS client now correctly handles PAUSED and INTERRUPTED job statuses.

​

API & Services

• Fine-tuning and serving accept a registry model tag alongside the legacy keyword: Newton fine-tuning, Newton batch serving (activity-detection, manual-generation), and Omega fine-tuning plus machine-state jobs all run dual-mode — a job that sends no tag behaves exactly as before. Profile-delivered config takes precedence over the legacy model_variant/presets lookup at a single branch point, and serving a promoted Omega head now loads directly from the registry bundle instead of the legacy fine_tune_checkpoint port.
• Create Job, Get Job, and List Jobs responses now return workflow_key:These responses now carry the workflow_key property with the same value as pipeline_key, which is retained unchanged for backward compatibility. workflow_key is the preferred name going forward. The pipeline registry responses keep only pipeline_key. This completes changes introduced in v1.1.7.
• Lens v2 supports live RTSP cameras again: A new RTSP pipeline drives a live camera through the same register → buffer → submit path as the file-video pipeline, wire-compatible with the legacy v1 stream configuration (rtsp_url, target_frame_rate_hz, target_image_size). Each connection gets a dedicated decode thread handing frames over a drop-oldest queue, so a stalled consumer keeps the freshest frames rather than compounding latency, and camera drops reconnect with exponential backoff. Session destroy is now honored by all four reader loops within about a second, so a destroyed session stops reconnecting instead of burning its backoff budget.
• JOS client supports the fine-tune API and KNN fine-tune jobs: The client can submit Fine-Tuning API jobs and Omega KNN preparation jobs directly.

​

Console & UI

• Job detail pages converge on one shared layout: Job detail pages across most of the UI have converged on a single design. Progress moved out of the tab strip to the top of the page as a “now” block, so the status banner no longer duplicates the progress bar for healthy running or cleanly succeeded jobs. Config moved from a rail card into its own full-width tab. Fine-tuning jobs have not yet adopted the new look.
• Simplified and improved the profile and API key UI: /profile now hosts profile details and API-key create/revoke together on one screen.

​

Security & Authentication

• Platform admins can deactivate and reactivate users from the console: Two new superadmin endpoints and matching buttons on the org members table close the gap that previously forced a superadmin to fall back to CLI plus port-forward plus master key. Actions are logged against the specific user’s user_id rather than a generic "admin" actor, and a superadmin cannot deactivate their own account. Reactivation deliberately does not restore revoked API keys — that would defeat the offboarding cascade — so users mint fresh keys via SSO.
• RTSP URL handling is hardened: The URL scheme is now allowlisted to rtsp/rtsps. Loopback and link-local literal IPs are rejected to close off the cloud metadata endpoint and localhost side channels, while private LAN ranges stay allowed since VPN-reachable on-premises cameras are the primary real source. URL credentials are redacted from every log line and emitted event.

​

Bug Fixes

• Batch-creation errors now include details about how to resolve the problem: About 20 strings across the create modal, file upload, and file pickers were rewritten so every message names the problem and the resolution, dropping developer jargon like raw JSON pointers and status codes.
• Long agent names no longer hide the status badge: The card header’s flex wrapper is a grid item defaulting to min-width:auto, so it refused to shrink and pushed the badge past the card’s clipped edge. Adding min-w-0 lets the title truncate instead, and a title attribute makes the full name readable on hover.
• Fine-tuning metrics reach the console again: The metric produced during MSJ inference over the fine-tune job was not saved alongside the checkpoint, so it never propagated to the UI.

​

API & Services

• Fine-tuned checkpoints can now be promoted directly to fine-tuned models with base model tracking. The base model tag is resolved and recorded at dispatch time, and both the checkpoint promotion and the resulting model responses now expose base model and tag information.
• Fine-tuning jobs no longer report success after an internal failure. Previously the worker could swallow an error and mark a failed fine-tuning job as successful, which is now corrected.
• Batch inference now accepts a workflow_key parameter, deprecating the older pipeline_type/pipeline_key fields. This simplifies how batch jobs specify which workflow to run.

​

Console & UI

• The Console sidebar was collapsed into an icon rail. The refresh also adopted secondary call-to-action alert actions from the updated design system.
• The Console service now validates its environment configuration on startup. Misconfigured deployments fail fast with a clear error instead of surfacing confusing runtime issues later.

​

Security & Authentication

• A wave of Critical/High severity CVEs has been remediated across non-PDM build surfaces. The fix updates vulnerable dependencies as part of an ongoing security hardening effort.
• IAM now cascade-revokes a user’s API keys when their account is deactivated. A companion CLI command was added so operators can trigger the same revocation workflow manually.

​

Bug Fixes

• Role checks on checkpoint promotion and model deletion were tightened. This closes a gap that could allow insufficiently privileged users to perform these actions.
• Z-score computation for flatlined channels in omega_1_5 is now stable across different frameworks and hardware. This fixes inconsistent results that previously varied depending on the runtime environment.

​

API & Services

• Lens Runner v2 is now enabled by default.
• GPQ now distinguishes cancelled queries from failed ones and fixes its default max_frames setting.

​

Console UI

• The Console UI underwent general normalization to bring a more cohesive feel to the interface.
• Batch inference jobs in the Batch Manager can now take a checkpoint as an input. This is shown using distinct UI treatment on the job’s input files tab.

​

IAM & Authentication

• Organization-admin and platform-admin capabilities have been added to IAM, gated by new JWT-scoped middleware. A new route group requires a JWT with sub_type=user and role=admin, unblocking Console pages for full organization member and settings management as well as a platform-wide super-admin grant and bulk member creation.
• The owner_user_id predicate is now enforced on files, jobs, lenses, and sessions, scoping resource visibility to the requesting user. This continues our work on per-user resource scoping and applies the same visibility predicate consistently across the data, JOS, and lens services.

​

Fine-Tuning Service

• A new v0.6 fine-tuning API has been introduced to JOS, providing a user-facing interface for creating and managing model training jobs. The API is mounted at /v0.6/fine_tuning and supports Newton and Omega model discrimination, cursor-paginated job listings, checkpoint endpoints, and per-model hyperparameter schemas. It maps high-level fine-tuning requests onto internal JOS training jobs and mirrors the OpenAI fine-tuning job object shape in its responses.
• The dedicated fine-tuning runner node has been retired and replaced by the general-purpose training worker infrastructure. This consolidates fine-tuning execution through the standard worker path and removes the now-obsolete standalone runner component from the platform manifest. The Fine-Tuning Node API has been deprecated and is no longer available for use.
• A fine-tuning dashboard has been added to the console for creating, managing, and monitoring training jobs. The page is modeled after the existing Batch Manager and includes a job list/grid view and a create-job modal scoped to training pipelines. Batch Manager pipeline listings have been updated to filter out training pipelines so the two sections remain distinct.

​

Job Orchestration Service

• Pause and Resume endpoints have been added to JOS along with new Paused and Interrupted job states. Cancel has been made a terminal transition (the Cancelled state is now final), and Retry now returns a 501 directing callers to Pause instead. The new Interrupted state covers resumable worker failures separately from the non-resumable Failed state, and resume is permitted from Interrupted.
• Retry endpoint has been disabled. The Retry Job endpoint has been deactivated for the time being. Using it results in a 501 Not Implemented error. This endpoint will return with a future release.
• Completed Kubernetes Jobs are now torn down immediately upon completion to reclaim node disk space. Previously only failed jobs triggered teardown, leaving model weight caches (~16 GB per pod) on GPU nodes for up to an hour, which caused intermittent “Insufficient cache space” failures for subsequent fine-tuning jobs on single-node GPU pools.
• Upfront validation for JSONL and CSV job inputs has been added, checking files before job creation rather than at runtime. File extension is now checked against the text/plain MIME type, and input selection validates types at the point of selection.
• A job outcome field has been added to JOS, and all jobs that finish execution are now marked Completed regardless of whether individual inputs had processing errors. This separates jobs that completed their run from jobs that failed to run at all, giving a more accurate picture of job health.
• Processed bytes are now included in the progress summary returned with each job response. This data powers the byte-based progress display in the Batch Manager job listing page.
• Status information for job outputs has been added to JOS job responses.

​

Console UI

• The Batch Manager has been significantly redesigned, introducing outcome modeling, a refreshed sidebar, a new job-details page, and byte-based progress tracking. Banner-style callouts for job outcomes (succeeded, partially processed, failed, cancelled), and byte-level progress from the server are now surfaced in the job list view.
• Pause and Resume buttons have been added to the Batch Manager UI, allowing users to suspend and continue in-progress jobs. The Cancel CTA has been restored alongside the new controls so all three lifecycle actions are available from the same interface.
• The “Task Type” label in the Batch Manager UI has been renamed to “Workflow Key” for clarity.
• Byte-level progress information on the Batch Manager job listing page has been fixed to display accurate values.

​

IAM & Authentication

• Bulk-revocation of all active API keys for one or more orgs has been added to the IAM service and CLI. A new master-key-protected endpoint (POST /v1/iam/orgs/{org_id}/keys/revoke-all) performs a single atomic update and emits per-key and summary audit events; the atai iam org revoke-keys CLI command defaults to a dry-run preview and requires --execute to apply changes.
• Structured audit events have been added across the full OIDC login flow in the console. Every meaningful state transition — authorize start, callback received, state validation failure, KC/IAM exchange failures, and the existing success/failure events — now emits a structured event with a shared flow_id UUID that correlates all steps for a single login attempt. No raw secrets or token values are logged.
• The owner_user_id column has been added to files, lenses, sessions, and JOS jobs to track resource ownership at the user level.

​

API & Services

• The Jobs Orchestration Service (JOS) now supports chunked file outputs, exposing a single logical output file per input regardless of how many segments a worker produces. Worker pods tag each segment result with a chunk_of field containing the chunk index, parent filename, and a final-chunk flag. Output URIs are rewritten to the authenticated /files/download endpoint, and files carry a job_output.status attribute (partial while in progress, complete when done) so users can track job state from the file library.
• JOS now detects and reports stuck jobs through a new background sweep and an OTel jobs_stuck gauge. A job is now correctly marked RUNNING only when its first pod becomes Ready, so jobs queued without available GPU remain in ADMITTED rather than prematurely appearing active to users. The sweep detects two conditions — a job stuck in ADMITTED for over an hour and a RUNNING job with no emitted event or progress for over an hour — and exposes both counts via the new metric.
• JOS workers now exit gracefully on job cancellation by catching SIGTERM, checkpointing the current segment, and terminating cleanly. The pod termination grace period has been raised from 30 seconds to 2 minutes by default to give workers sufficient time to complete a checkpoint before Kubernetes forcibly terminates the pod. The grace period is configurable per deployment.
• A manual-generation JOS pipeline has been added for producing structured procedural documentation from video. The pipeline performs audio-grounded batch inference and is available as a new task type configured via a JOS seed deployed to the environment.

​

Security & Authentication

• API keys can now be created with an optional TTL, and expired keys are automatically marked in the database on first use after expiry. A new --ttl-days flag on the atai iam key create and atai iam user key create CLI commands computes an expires_at timestamp at creation time and passes it through to the IAM service. When an expired key is rejected, its status transitions from active to expired. No schema migration is required as the expires_at column already existed on the api_keys table.

​

API & Services

• Chunked files can now be downloaded via the existing /files/download endpoint, with content streamed directly from S3 in chunk order: The download path detects whether a file is chunked and assembles the response as a streaming byte sequence across all stored S3 objects, with no need for a separate endpoint or client-side logic.
• A new SegmentedInferenceSession abstraction has been added to the JOS worker client, enabling automatic checkpointing, progress reporting, and resilient resume for batch inference jobs: Workers now iterate over “segments” (subsets of input files) rather than whole files, and the session handles checkpointing between segments, emitting file-started/completed events, and resuming from the last checkpoint on restart. Segment size is configurable with defaults of 50 MB of input or 10 minutes, whichever comes first. The Nano inference worker and the inference phase of the machine state job have been updated to use this abstraction.
• Stale file uploads stuck in the UPLOADING state are now automatically cleaned up: A configurable scheduler (default: every 6 hours) aborts any upload older than a threshold (default: 24 hours) using the same logic as the /abort endpoint.
• A new DatasetList port mode and a PortMode::Checkpoints type have been added to JOS to support dataset listing and checkpoint upload workflows: These additions enable workers to declare input/output ports that carry dataset lists and model checkpoints respectively, broadening the range of job types that can be expressed natively in JOS config. Checkpoint files are now staged under a non-tmpfs folder to avoid memory pressure.

​

Inference & Model Updates

• The C2.6 model is now available with a new vLLM inference backend that delivers improved throughput over previous transformer-based inference: Multi-image support has been added to Direct Query and the Nano engine, and GPU-specific YAML configurations allow fine-grained tuning per model and hardware combination.
• GPU memory budget calibration for C2.5 and C2.6 has been improved with a more accurate adaptive probe sweep across batch size and shape dimensions: The calibration now accounts for padding-aware batch peaks, shrinks the memory budget incrementally after each OOM, and warms up the engine at max probe shapes before calibrating. This reduces spurious OOM events during production batch inference workloads.
• UFM training has been extended with variable-K Stage 1+2 resolution, pad_mode=interpolate retraining, Stage 3 sweep plumbing, and several stability fixes: A deterministic eval fix and a PEP 479 StopIteration correction were also merged, along with pinning flash-linear-attention to avoid a ~3× training slowdown introduced by a dependency update. The UFM builder and model directories were refactored to align with the factory pattern introduced in the updated Nano engine.

​

Observability & Telemetry

• OTel SQL tracing has been extended to cover database transactions via a new TracedTransaction wrapper, and rolled out to the Lens API and Data services: This completes full OTel tracing coverage for data_service and lens_api_service; iam_service and jos_service will follow in subsequent releases.
• New Python OtelMetricsBase and OtelWorkerMetricsBase base classes have been introduced, and the first batch of active platform services has been migrated from Prometheus to OpenTelemetry metrics: The new base classes use the global MeterProvider directly, dropping the per-service CollectorRegistry and update() chain used by the legacy Prometheus classes. Services migrated in this release include api_service, lens_service, and gpq_service; the previous service_info gauge is replaced by OTel resource attributes.
• Prometheus business metrics in data_service, iam_service, jos_service, lens_api, health_service, and registry_service have been migrated to the OTel push model: This brings the Python service telemetry pipeline in line with the Rust services and enables unified metric collection through the OTLP exporter. Per-request org_id propagation has also been added to OTLP logs, and per-signal OTLP endpoint overrides are now supported for routing traces, metrics, and logs to different collectors.
• New Grafana dashboard panels have been added for JOS job counts, node pool sizes, total file counts and bytes, and the sync has been improved to include fixes for overcounting: The panels provide visibility into per-org file storage volume and JOS queue depth and were synchronized with the cloud environment. A separate fix corrects an over-counting bug in the file count and bytes panels introduced during the initial rollout.

​

Console & UI

• Filename search has been added to the files dashboard, backed by new database indexes for performant queries: Users can now filter the file list by partial filename match directly in the UI, and the underlying data service database has been updated with the necessary indexes to keep query latency low even for large file tables.
• The Console login page, homepage, and top navigation have been migrated to the Archetype design system: Light/dark mode persistence is preserved.
• The Console dashboard and workbench pages have been migrated to the design system: Dashboard pages now use design system primitives, and the workbench has been updated while keeping all existing functionality intact.

​

Bug Fixes

• vLLM batch job result ordering and crash recovery have been fixed to ensure JSONL output is always in input order and that fatal engine failures are handled cleanly: When async vLLM workers complete out of order, results are now staged by submit position and flushed in contiguous runs, keeping output aligned with the original input. Per-record failure messages are sanitized to “inference error” so internal exception details never reach user-downloadable files. When the vLLM engine encounters a fatal error (such as CUDA out-of-memory), the worker now emits an inference.engine_dead event, cancels in-flight work, and exits cleanly so Kubernetes restarts the pod and JOS resumes from the last checkpoint.
• A TOCTOU race condition and stale counter bug in chunked file writes have been fixed, and chunk data and S3 objects are now properly cleaned up on file deletion: Concurrent appenders previously could race on chunk_index assignment; the fix ensures all writes happen within a single locked transaction. Deleting a chunked file now removes all associated file_chunks rows and their corresponding S3 objects, preventing orphaned storage.

​

API & Services

• file_type filter added to the list-files endpoint: A new query parameter allows callers to filter the file listing by MIME type or file type category, making it easier to retrieve only the files relevant to a specific workflow.
• OpenAPI schema for /query expanded with full request/response documentation: The OpenAPI spec for the /query endpoint now includes complete request and response schemas, improving SDK generation quality and developer documentation.
• ATAI_CA_BUNDLE_PATH environment variable renamed to ATAI_CA_BUNDLE: The legacy environment variable name was simplified for consistency. Deployments using the old name will need to update their configuration.
• Per-org file count and byte total exposed as Prometheus metrics (PLDEV-784): Two new Prometheus gauges track total file count and aggregate storage usage per organisation, enabling capacity planning and billing dashboards.
• The jos seed apply command now bundles seed YAMLs (PLDEV-909): The jos seed apply CLI command was updated to ship seed YAML files directly with the binary, so applying default data seeds no longer requires a separate file distribution step.
• Example automatic migration scripts added for 1.0.9 → 1.1.1 and 1.1.1 → 1.1.2: Reference migration examples were committed to the repository so operators have a concrete starting point for upgrading existing deployments across these version boundaries.

​

Observability & Telemetry

• Full OpenTelemetry OTLP distributed tracing stack added to all Rust services: The atai_telemetry crate now wires a complete three-signal OTel pipeline — distributed traces, log records with trace/span correlation, and push metrics — all exported via OTLP/gRPC when OTEL_EXPORTER_OTLP_ENDPOINT is set. HTTP spans follow OTel semantic conventions via axum-tracing-opentelemetry, with W3C traceparent extraction and propagation across all four Rust HTTP services. When the endpoint is unset, the crate behaves exactly as before with zero OTel overhead.
• OTLP log bridge and trace improvements added to console_2_service: This PR wires the OTLP log bridge into console_2_service and improves trace context handling, including always-on W3C trace ID generation and X-Request-Id reflection in responses. Static asset requests are excluded from tracing noise, and root HTTP span naming now uses METHOD /route/[param] with SvelteKit route IDs.
• init_tracer() added to atai_py and wired into initialize_service_logging: Python services can now initialize OpenTelemetry tracing with a single call, automatically reading OTEL_EXPORTER_OTLP_ENDPOINT and configuring the tracer pipeline (PLDEV-801).
• PLATFORM_VERSION is now propagated through logs and metrics: The platform version string is attached as a resource attribute on all telemetry signals, making it easier to correlate observability data with specific release versions.
• Legacy x-trace-id header dropped in favor of the OTel traceparent standard: The proprietary x-trace-id request/response header has been removed and all tracing correlation now relies on the W3C traceparent header (PLDEV-797). Any tooling or dashboards relying on x-trace-id will need to be updated.
• atai_telemetry_reqwest crate adds OTel HTTP client spans via reqwest-middleware: Outgoing HTTP calls made through reqwest now produce properly attributed OTel client spans (PLDEV-798), enabling end-to-end trace stitching for calls leaving Rust services.

​

Security & Authentication

• JWT exchange endpoint and JWKS added to the IAM service: A new POST /v1/iam/exchange endpoint accepts an API key and returns a signed RS256 JWT containing Archetype claims (subject, org ID, role, auth method), laying the foundation for moving from per-request API key validation to JWT-based auth. A companion GET /v1/iam/.well-known/jwks.json endpoint serves the RSA public key so downstream services can validate tokens locally. The existing /v1/iam/authenticate endpoint is unchanged for backward compatibility.
• UPLOADING status is now exposed in the public file API (PLDEV-833): Previously the UPLOADING state was hidden from the public-facing file status endpoint; it is now surfaced so clients can accurately track in-progress uploads.

​

Data Integrity & Uploads

• End-to-end server-driven checksum verification added to the direct upload flow (PLDEV-663): The server now selects the checksum algorithm at upload initiation and returns it in the InitiateUploadResponse; clients can optionally compute and submit a whole-file CRC32C in CompleteUploadRequest. If provided and mismatched, the file is marked CORRUPT and HTTP 422 is returned; if absent, the upload succeeds without verification, preserving backward compatibility. S3 CreateMultipartUpload is called with ChecksumType: FullObject so a single whole-object CRC32C is stored and retrievable via HeadObject.

​

Inference & Model Features

• vLLM engine support added along with c26 improvements: The vLLM inference engine backend was integrated, including model weight caching and related configuration changes for the c26 hardware generation.
• Stage 3 task classification expanded with MoteStrain and PAMAP2 datasets for UFM (fixed and variable): The UFM task classification pipeline now supports MoteStrain and PAMAP2 benchmarks across both fixed and variable input configurations.
• Jobs now fail explicitly if any input fails (PLDEV-940): Previously a job could silently succeed even if one or more of its inputs errored; the job runner now propagates input failures and marks the overall job as failed.

​

Console & UI

• Autocomplete label suggestions added to the n-shot file picker (PLDEV-715): As users enter class labels across n-shot files, a session-scoped vocabulary is built up and previously used labels are surfaced as autocomplete suggestions on subsequent inputs. The dropdown supports keyboard navigation (Arrow Up/Down, Enter, Escape), mouse selection, and auto-scrolls the active suggestion into view.
• Error tooltip in the console stays visible longer and supports text copying: The tooltip that appears on API or service errors now remains on screen long enough to read and can have its content copied, improving the debugging experience for users (PLDEV-786).
• Content-aware column widths applied to batch manifest tables: Columns in batch manifest views now size themselves based on their content rather than using fixed widths, improving readability for a wide range of payload shapes.
• Progress chart tooltip is now flipped when near the right edge (PLDEV-889): The tooltip on progress charts was being clipped when the cursor was near the right boundary; it now flips to the left to stay fully visible.
• “Pipeline” label renamed to “Task Type” throughout the console (PLDEV-905): The UI label used to describe processing pipelines has been standardized to “Task Type” for consistency with the rest of the product terminology.
• MSJ broken image display fixed: A regression that caused broken image previews in the multi-sensor join UI was resolved.
• Wrong progress counter in MSJ fixed (RES-272): A display bug that caused incorrect progress percentages to appear in the MSJ task view was corrected.

​

New Features & Improvements

• Added direct-to-cloud file upload support to the Rust and Python SDKs (PLDEV-535, PLDEV-16): Both the Rust and Python SDKs now support uploading files directly to cloud storage via presigned S3 URLs, bypassing the data service proxy. The Rust SDK introduces a new builder API (UploadBuilder) as the default path, with the proxy path still available via .using_proxy(); the Python SDK makes the direct path opt-in via use_proxy=False. Both implementations support concurrent multipart uploads with configurable worker counts, per-part retries with exponential backoff, progress callbacks, and cancellation. This enables uploads well beyond the proxy’s previous 500 MB size limit.
• Added resumable upload support to the Rust and Python SDKs (PLDEV-551, PLDEV-778): Clients that fail mid-upload can now resume from where they left off without re-uploading already-completed parts. A new server-side checkpoint endpoint stores completed part tokens, and the initiate call accepts a resume_if_started flag to reuse an in-progress upload for the same file. On the Rust SDK, resuming is enabled via .with_resume(true) on the upload builder, with checkpointing on by default; on the Python SDK, resuming is enabled by passing allow_resume=True to upload(), with checkpointing also on by default. If a progress callback is set, it will be called once with the already-uploaded byte count when a resume occurs.
• Increased the maximum upload file size to 250 GB: The platform-wide maximum file size for uploads has been raised to 250 GB to accommodate large dataset and model artifact transfers.
• Added a session validation step to the Workbench (PLDEV-195, PLDEV-600): The Workbench now sends an explicit session.validate event at the start of every lens session before entering the active streaming state. If validation fails, users receive a clear error notification even when the session log panel is collapsed, and error log entries are highlighted in red for quick visual identification. This addresses a recurring issue where heartbeat timeouts would cause sessions to continue in a degraded state without surfacing clear feedback to users.
• Added a read-only selected file name field to the Workbench lens tray (PLDEV-609): The lens tray settings panel for the Activity Monitor and Machine State lenses now displays the name of the currently selected input file below the model version field. When no file has been chosen, the field shows “Not selected.”
• Limited CSV table and graph rendering to 10,000 rows in the Workbench to prevent freezes on large datasets (PLDEV-186): When an uploaded CSV file exceeds 10,000 rows, the Workbench now truncates the preview display and shows a banner informing users of the total row count and directing them to the API for full data access. CSV data is now truncated at the line-split stage before full parsing occurs, preventing memory pressure from very large files.
• Improved Workbench output panel autoscroll behavior (PLDEV-350): The Workbench output panel now auto-scrolls to the latest response by default but pauses when the user manually scrolls up to review earlier results. A NEWEST button appears when the user has scrolled away from the bottom, and clicking it jumps back to the latest output and resumes auto-scrolling. Autoscroll state is reset at the start of each new session.
• Added drag-and-drop bulk file upload to the File Manager (ATAI-2938): Users can now upload files to the File Manager by dragging and dropping them directly onto the file list page, or by using a new modal-based upload dialog triggered from the “Add files” button. The upload dialog shows per-file progress, supports individual file cancellation, and retains failed upload placeholders in the list so users can see which uploads did not complete.
• Added the Batch Manager to the Console with live job creation, listing, and detail pages (PLDEV-575, PLDEV-576, PLDEV-604, PLDEV-689, PLDEV-695): A new Batch Manager page is available in the Console, allowing users to create and monitor batch jobs submitted to the Job Orchestration Service, and to download job artifacts. See the Batch Manager documentation for details.
• Added Python JOS clients for API access and job container use (PLDEV-348): Two new Python clients for the Job Orchestration Service have been added. JosApiClient covers all 20 REST API endpoints (jobs, components, pipelines) with an async-first design and structured error handling. JosWorkerClient is for use inside JOS-managed pods and provides InputPort/OutputPort abstractions for reading JSONL manifests from S3, uploading outputs, reporting progress via Redis, and saving and restoring checkpoints.
• Synced CSV window_size and step_size from the Workbench lens tray to the input stream config (PLDEV-629): When a user updates the window_size or step_size fields in the CSV lens tray, the values are now propagated to the underlying input stream config so that inference window boundaries correctly reflect the configured parameters.
• Added structured error responses to the data service (PLDEV-591): The data service now returns structured, machine-readable error response bodies across its endpoints, replacing unstructured text errors.

​

Bug Fixes

• Fixed lens worker crash loops from missing model_parameters and stale Redis events (PLDEV-610): A cascading failure was identified where missing model_parameters in a lens config caused a KeyError crash on worker nodes, and stale Redis queue events from crashed sessions caused nodes to re-enter crash loops on restart. Five targeted fixes prevent the KeyError, drain stale Redis events on node restart, skip unknown-session events, and garbage-collect stuck sessions.
• Fixed narrator memory not being initialized for direct model.query calls without a video stream (PLDEV-632): When model.query was called without a preceding stream.start event — for example, for non-video file inputs — the narrator’s internal memory and buffer fields were uninitialized, causing failures.
• Fixed deleted files returning stale data from the Files API (PLDEV-614, PLDEV-605): The GET /files/metadata/{id} endpoint now returns 404 Not Found on deleted files instead of the old record with an “unknown” status, and the download endpoint also returns 404 instead of 400 on deleted files. Re-uploading a file to a previously deleted file key is now permitted.
• Fixed multipart upload completion not accepting unsorted parts (PLDEV-670): The /files/uploads/{id}/complete endpoint no longer requires the uploaded parts list to be submitted in sorted order.
• Fixed the window_size and step_size values not being returned by the console-2-service (PLDEV-600): The backend was not including window_size and step_size in its responses, causing the lens tray to display stale or missing values after a session was initialized.
• Fixed the console CSV config tooltip width and wrapping: The tooltip for CSV configuration settings in the console was overflowing its container; width and text-wrapping constraints have been applied.
• Fixed the console-2-service to correctly apply schema default values for config templates: The console frontend now correctly applies schema-defined default values when rendering config template fields, rather than leaving them empty.

​

New Features & Improvements

• Added the Fine-Tuning Node to perform fine tuning jobs on dedicated GPUs: The new Fine-Tuning Node offers an API to create, manage, and monitor fine-tuning jobs for your organization. Each Fine-Tuning Node runs a fine-tuning job on its assigned GPU, acting like a worker that trains a model using the provided dataset and configuration. This produces a fine-tuned model and training metrics.
• Added the Batch Manager to the Console with live job creation and listing (PLDEV-575, PLDEV-576): A new Batch Manager page is now available in the Console (behind the CONSOLE_FLAG_SHOW_BATCH_MANAGER feature flag), allowing users to create and monitor batch jobs submitted to the Job Orchestration Service. The Batch Manager supports creating jobs with file input selection and YAML/JSON config validation, displays submitted jobs with status badges and relative timestamps, and fetches job and pipeline data server-side.
• Added a session validation step to the Workbench (PLDEV-195, PLDEV-600): The Workbench now runs an explicit session.validate event at the start of every lens session before entering the active streaming state. If validation fails, users receive a clear error notification even when the session log panel is collapsed, and error log entries are highlighted in red for quick identification.
• Added a read-only selected file name field to the Workbench lens tray (PLDEV-609): The lens tray settings panel for the Activity Monitor and Machine State lenses now displays the name of the currently selected input file below the model version field. When no file has been chosen, the field shows “Not selected.”
• Limited CSV table and graph rendering to 10,000 rows in the Workbench to prevent freezes on large datasets (PLDEV-186): When an uploaded CSV file exceeds 10,000 rows, the Workbench now truncates the preview display and shows a banner informing users of the total row count and directing them to the API for full data access.
• Improved Workbench output panel autoscroll behavior (PLDEV-350): The Workbench output panel now auto-scrolls to the latest response by default but pauses scrolling when the user manually scrolls up to review earlier results. A “NEWEST” button appears when the user has scrolled away from the bottom, allowing them to jump back to the latest output and resume auto-scrolling. Autoscroll state is reset at the start of each new session.
• Added drag-and-drop bulk file upload to the File Manager (ATAI-2938): Users can now upload files to the File Manager by dragging and dropping them directly onto the file list page or by using the new modal-based upload dialog. The upload dialog supports multiple concurrent file uploads, individual cancellation, progress tracking per file, and persists failed upload placeholders in the file list so users can see which uploads did not complete.
• Implemented the Machine State Job (RES-226): A new batch job for machine state classification has been added, built on the fine-tuned Omega 1.3 model. The job can run on both CPU and GPU, processes sensor CSV data, and supports n-shot input files for healthy and faulty reference examples.
• Synced CSV window_size and step_size from the Workbench lens tray to the input stream config (PLDEV-629): When a user updates the window_size or step_size fields in the CSV lens tray configuration, the values are now propagated to the underlying input stream config so that the inference window boundaries correctly reflect the configured parameters.

​

Bug Fixes

• Fixed narrator memory not being initialized for direct model.query calls without a video stream: When model.query was called without a preceding stream.start event (e.g., with non-video file inputs), the narrator’s memory and buffer fields were uninitialized, causing the lens service to fail.
• Fixed deleted files returning stale data from the Files API (PLDEV-614, PLDEV-605): The GET /files/metadata/{id} endpoint now returns a 404 Not Found response when called on a deleted file, rather than returning the old file record with an “unknown” status. The GET /files/download/{id} endpoint similarly now returns 404 on deleted files instead of a 400 error. Additionally, the Files API now allows re-uploading a new file over a previously deleted file’s key.
• Fixed the console CSV config tooltip width and wrapping: The tooltip for CSV configuration settings in the console was overflowing its container; width and text-wrapping constraints have been applied.