Atlas Stream Processing Changelog

Enables project-level streams auditing by default and removes the Streams Auditing toggle from the Database & Network Access > Security > Advanced page in the Atlas UI. Audit logs are now always-on and non-deletable to meet security and compliance expectations. To learn more about Atlas Stream Processing auditing, see Atlas Stream Processing Access and Auditing.

Adds a real-time status (Ready, Pending, Failed, Deleting) to the Atlas UI when you view connections in the connection registry.

Adds support for AWS Kinesis connections as source and sink options when you Create a Stream Processor in the Atlas UI.

Adds support for the Download Operational Logs for One Atlas Stream Processing Workspace endpoint in the Atlas Administration API to download operational logs for Atlas Stream Processing workspaces or stream processors.

Fixes an issue that prematurely marked streams connections as Ready while the underlying Atlas cluster was still provisioning.

Fixes an issue where the modifyStreamProcessor action was unavailable for role assignment.

Improves error handling for the operations to start, stop, and delete stream processors.

Improves validation for Azure Event Hub connections to ensure only valid Namespace ARM URIs are accepted.

Adds support for Avro serializarion.

Adds support for integrations with Confluent Schema Registry.

Allows writing over an HTTPS connection using $https as a final stage in a pipeline.

Adds support for AWS Kinesis as a source ($source stage) and as a sink ($emit stage).

Updates data transfer pricing to reflect region-specific rates for both the stream processor and destination region. To learn more, see stream processor pricing.

Updates pricing for SP10 and SP30 tier stream processors from the legacy worker-based model to the per-processor pricing model. To learn more, see stream processor pricing.

Replaces the previous instance model with Atlas Stream Processing Workspaces. To learn more, see Stream Processing Tiers.

Adds new stream processor tiers: SP2, SP5, and SP50.

Adds support for OIDC with JWT Authentication for Apache Kafka connections.

Adds support to create a stream processor pipeline in the Atlas UI.

Adds support for the AWS S3 private link connection type.

Adds the Test Connection button in the Connection Registry UI to initiate a connectivity test for each connection.

Adds the tenant_name, project_name, and org_name fields to Atlas Stream Processing metrics, including those exported to Datadog.

Adds lifetime cache hit/miss statistics to the stats.operatorStats of the $cachedLookup stage.

Adds support for the Stream Processor State metric in the Atlas Datadog integration.

Adds the $function operator, allowing for Javascript execution in stream processing pipelines. To learn more, see $function (Stream Processing).

Adds the $cachedLookup operator allowing for $lookup results to be cached for a specific TTL to improve the performance of slowly changing data. To learn more, see $cachedLookup.

Adds the ability for the $emit stage to send null values to Kafka topics to support compacting Kafka Topics by using $emit.config.tombstoneWhen = boolean. To learn more, see $emit Aggregation Stage (Stream Processing).

Adds support for Cross Atlas Project for $source and $merge. To learn more, see the Replicate Change Stream Events Across Atlas Clusters section in the $merge (Stream Processing) reference.

Adds support for $source to support Initial Sync when reading from Atlas collections. To learn more, see MongoDB Collection Change Stream.

Adds support for Stream Processing Workspaces to deploy on Google Cloud. To learn more, see Google Cloud (GCP) Stream Processing Regions.

Adds support for $merge to be able to delete documents in collections. To learn more, see $merge (Stream Processing).

Adds support for Datadog Metrics. To learn more, see Datadog Metrics.

Adds support for AWS S3 connections.

Adds a new $sessionWindow Atlas Stream Processing pipeline aggregation stage that specifies a session window for aggregation of data. This allow you to run a pipeline on each "session" of activity in an input stream. To learn more, see $sessionWindow.

Supports Azure-hosted Confluent Kafka clusters via Private Link connection.

Adds a new $externalFunction Atlas Stream Processing pipeline stage that triggers processes in a specific AWS Lambda resource. To learn more, see $externalFunction.

Supports creating $https connections in the Atlas UI.

Adds the parallelism field to $merge. The field specifies the number of threads to which to distribute write operations, which improves performance.

Allow you to create additional alerts: Output Message Count, DLQ Message Count, Kafka Lag, and Change Stream Delay. To learn more, see Atlas Stream Processing Alerts.

Supports the createUUID expression that takes no arguments and returns UUID BSON type values in Stream Processors.

Adds a new configuration for Window operators to support processingTime in addition to eventTime. To learn more, see Processing Time.

Adds the $meta expression that returns an object containing all streaming metadata for a document.

Adds the parseJsonStrings field to the $https operator, allowing it to parse JSON Strings returned from an API call. To learn more, see $https.

Allows you to add a Kafka Private Link Connection for AWS MSK clusters.

Allows you to deploy Stream Processing Workspaces on AWS us-east-2. To learn more, see Atlas Stream Processing Regions.

Supports the $currentDate expression that returns the system time of your stream processing workspace each time Atlas Stream Processing evaluates it.

Supports reading JSON documents with embedded file signatures (magic bytes).

Fixes an issue that prevented the configuration of hopping windows with hopSize greater than interval.

Changes the executionTimeSecs stat to executionTimeMillis. To view this stat, invoke the sp.processor.stats() command.

Changes the buffering duration for $emit Aggregation Stage (Stream Processing) to Apache Kafka sinks from 1000 milliseconds to five milliseconds.

Supports VPC Peering with Confluent on AWS.

Supports using the Atlas Administration API to edit stream processors.

Supports AWS PrivateLink with Confluent.

Supports Azure Private Link with EventHubs.

Increases Apache Kafka Timeout from 10 seconds to 30 seconds.

Always retries processors when the target cluster is in the REPAIRING state.

Fixes the partitionIdleTimeout after the checkpoint restore.

Adds more watermark and window information to verbose stats.

Ensures calls to produce messages to Apache Kafka don't hang indefinitely, even when the producer queue is full.

Supports ingesting multitopic Apache Kafka sources.

Updates Apache Kafka offsets for consumer groups to near-real time.

Supports the compression.type and acks settings in the Apache Kafka $emit configuration.

Supports collectionless $lookup syntax.

Standardizes the way Atlas Stream Processing serializes Apache Kafka errors into a string.

Increases the $emit and $merge dynamic expression limit to 1000.

Handles $lookup with unique connectionName inside window pipeline.

Improves paused cluster errors.

Supports converting from Binary data to various data types using $convert. This allows you to transform Kafka headers into specific data types as needed.

Stream processors that use Apache Kafka as a source expose per-partition watermarks in sp.processor.stats().

Fixes a bug where users were unable to create stream processing workspaces in certain regions through the API.