Limitations

The following limitations apply to Atlas Stream Processing:

• Atlas Stream Processing supports only at-least-once processing.

• Atlas Stream Processing doesn't support horizontal scaling.

• Atlas Stream Processing uses a single core for transformation pipeline stages, except for stages that can specify a parallelism value.

• The state.stateSize of a stream processor can't exceed 80% of the RAM available for its pod. For example, the maximum size of a stream processor in the SP30 tier, which has 8GB of RAM, is 6.4GB. If the state.stateSize of any of your stream processors is approaching 80% of its available RAM, consider stopping the processor and restarting it on a higher tier. If your stream processor already runs at the maximum tier enabled for your stream processing workspace, consider adjusting your stream processing workspace configuration to enable higher-tier stream processors.

  When a stream processor crosses the 80% RAM threshold, it fails with a Worker out of memory error. You can view the state.stateSize value of each stream processor with the sp.processor.stats() command. See View Statistics of a Stream Processor to learn more.

• An Atlas Stream Processing pipeline definition cannot exceed 16 MB.

• Only users with the Organization Stream Processing Admin, Project Owner, Project Stream Processing Owner or Atlas admin roles can use Atlas Stream Processing.

• For Apache Kafka $source stages, if the Apache Kafka topic acting as $source to the running processor adds a partition, the starting offset is determined by the auto_offset_reset property.

• For Apache Kafka $source stages are subject to stream processing workspace tier partition limits. If you exceed the partition limit of your stream processing workspace tier, the affected stream processor will fail. You must upgrade your stream processing workspace to support additional partitions.

• Atlas Stream Processing doesn't support connecting to an Apache Kafka consumer in the same consumer group as other consumers. Ensure any Kafka consumer used with Atlas Stream Processing runs in a dedicated consumer group.

• For Apache Kafka connections, Atlas Stream Processing currently supports only the following security protocols:

  • SASL_PLAINTEXT

  • SASL_SSL

  • SSL

  For SASL, Atlas Stream Processing supports the following mechanisms:

  • PLAIN

  • SCRAM-SHA-256

  • SCRAM-SHA-512

  • OAUTHBEARER

  For SSL, you must provide the following assets for your Apache Kafka system mutual TLS authentication with Atlas Stream Processing:

  • a Certificate Authority (if you are using one other than the default Apache Kafka CA)

  • a client TLS certificate

  • a TLS keyfile, used to sign your TLS certificate

• Atlas Stream Processing supports a subset of the Aggregation Pipeline Stages available in Atlas, allowing you to perform many of the same operations on streaming data that you can perform on data-at-rest. For a full list of supported Aggregation Pipeline Stages, see the Stream Aggregation documentation.

• Atlas Stream Processing doesn't support the aggregation variables $$NOW, $$CLUSTER_TIME, $$USER_ROLES, and $SEARCH_META.

• Atlas Stream Processing doesn't support writing BSON documents larger than 125 MB using the $emit stage to an AWS S3 bucket.

• You must use mongosh version 2.3.4 or higher to override an option on an existing stream processor using a mongosh method. For example, using sp.processor.start() to specify a tier for the processor you are starting.

  See Develop Stream Processors to learn more about managing a stream processor using mongosh.

• Atlas Stream Processing supports failover processors only for processors with the following configurations:

  • Atlas source and sink

  • Atlas source and Apache Kafka sink

• Atlas Stream Processing supports failover processors only for processors of tier SP10 or higher.

• For processors configured with failover processors, only one processor can be active at any given time. You can edit only the active processor.