> For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt

# Atlas Stream Processing Limitations

**Note:**

This page applies to both Atlas Infinite and Atlas Core.

The following limitations apply to Atlas Stream Processing:

## General

- Atlas Stream Processing supports only at-least-once processing.

- Atlas Stream Processing doesn't support horizontal scaling.

- Atlas Stream Processing uses a single core for transformation pipeline stages, except for stages that can specify a `parallelism` value.

- When you enable [vertical autoscaling](https://www.mongodb.com/docs/atlas/atlas-stream-processing/autoscaling.md#std-label-atlas-sp-autoscaling), you must set a `maxTier` either on the processor or as the stream processing workspace maximum tier size. If neither is set, the processor returns an error and doesn't start.

- You can't configure the CPU and memory thresholds that trigger vertical autoscaling. Atlas Stream Processing manages these thresholds internally.

- The `state.stateSize` of a stream processor can't exceed 80% of the RAM available for its pod. For example, the maximum size of a stream processor in the `SP30` tier, which has 8GB of RAM, is 6.4GB. If the `state.stateSize` of any of your stream processors is approaching 80% of its available RAM, consider stopping the processor and restarting it on a higher tier. If your stream processor already runs at the maximum tier enabled for your stream processing workspace, consider adjusting your stream processing workspace configuration to enable higher-tier stream processors.

  When a stream processor crosses the 80% RAM threshold, it fails with a `Worker out of memory` error. You can view the `state.stateSize` value of each stream processor with the `sp.processor.stats()` command. See [View Statistics of a Stream Processor](https://www.mongodb.com/docs/atlas/atlas-stream-processing/manage-stream-processor.md#std-label-atlas-sp-manage-processor-stats) to learn more.

- An Atlas Stream Processing pipeline definition cannot exceed 16 MB.

- Only users with the [`Organization Stream Processing Admin`](https://www.mongodb.com/docs/atlas/reference/user-roles.md#mongodb-authrole-Organization-Stream-Processing-Admin), [`Project Owner`](https://www.mongodb.com/docs/atlas/reference/user-roles.md#mongodb-authrole-Project-Owner), [`Project Stream Processing Owner`](https://www.mongodb.com/docs/atlas/reference/user-roles.md#mongodb-authrole-Project-Stream-Processing-Owner) or [`Atlas admin`](https://www.mongodb.com/docs/manual/reference/built-in-roles.md#mongodb-atlasrole-Atlas-admin) roles can use Atlas Stream Processing.

- You must use [`mongosh`](https://www.mongodb.com/docs/mongodb-shell.md#mongodb-binary-bin.mongosh) version 2.3.4 or later to override an option on an existing stream processor using a [`mongosh`](https://www.mongodb.com/docs/mongodb-shell.md#mongodb-binary-bin.mongosh) method. For example, using `sp.processor.start()` to specify a tier for the processor you are starting.

  See [Develop Stream Processors](https://www.mongodb.com/docs/atlas/atlas-stream-processing/manage-stream-processor.md#std-label-atlas-sp-manage-processor) to learn more about managing a stream processor using [`mongosh`.](https://www.mongodb.com/docs/mongodb-shell.md#mongodb-binary-bin.mongosh)

## Aggregation Pipeline

- Atlas Stream Processing supports a subset of the Aggregation Pipeline Stages available in Atlas, allowing you to perform many of the same operations on streaming data that you can perform on data-at-rest. For a full list of supported Aggregation Pipeline Stages, see the [Stream Aggregation documentation.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/stream-aggregation-stages.md#std-label-atlas-sp-aggregation)

- Atlas Stream Processing doesn't support the aggregation variables `$$NOW`, `$$CLUSTER_TIME`, `$$USER_ROLES`, and `$SEARCH_META`.

- Atlas Stream Processing doesn't support writing BSON documents larger than `125 MB` using the [$emit](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-emit.md#std-label-atlas-sp-agg-emit) stage to an AWS (Amazon Web Services) S3 (Simple Storage Service) bucket.

- The [$iceberg](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-iceberg.md#std-label-atlas-sp-agg-iceberg) stage supports a maximum of three partition transforms in the `partitionedBy` field.

## Kafka Connections

- For [Apache Kafka](https://kafka.apache.org/) [$source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-source.md#std-label-atlas-sp-agg-source) stages, if the [Apache Kafka](https://kafka.apache.org/) topic acting as [$source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-source.md#std-label-atlas-sp-agg-source) to the running processor adds a partition, the starting offset is determined by the [auto\_offset\_reset](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-source.md#std-label-atlas-sp-agg-source-syntax-kafka) property.

- For [Apache Kafka](https://kafka.apache.org/) [$source](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-source.md#std-label-atlas-sp-agg-source) stages are subject to stream processing workspace tier partition limits. If you exceed the partition limit of your stream processing workspace tier, the affected stream processor fails. You must upgrade your stream processing workspace to support additional partitions.

- Atlas Stream Processing doesn't support connecting to an [Apache Kafka](https://kafka.apache.org/) consumer in the same consumer group as other consumers. Ensure any Kafka consumer used with Atlas Stream Processing runs in a dedicated consumer group.

- For [Apache Kafka](https://kafka.apache.org/) connections, Atlas Stream Processing currently supports only the following security protocols:

  - `SASL_PLAINTEXT`

  - `SASL_SSL`

  - `SSL`

  For `SASL`, Atlas Stream Processing supports the following mechanisms:

  - `PLAIN`

  - `SCRAM-SHA-256`

  - `SCRAM-SHA-512`

  - `OAUTHBEARER`

  For `SSL`, you must provide the following assets for your [Apache Kafka](https://kafka.apache.org/) system mutual TLS authentication with Atlas Stream Processing:

  - a Certificate Authority (if you are using one other than the default [Apache Kafka](https://kafka.apache.org/) CA)

  - a client TLS certificate

  - a TLS keyfile, used to sign your TLS certificate

## Initial Sync

- [initialSync](https://www.mongodb.com/docs/atlas/atlas-stream-processing/sp-agg-source.md#std-label-atlas-sp-agg-source) doesn't support collections whose [\_id](https://www.mongodb.com/docs/manual/core/document.md#std-label-document-id-field) values are arrays, regular expressions, or undefined, which MongoDB doesn't support as `_id` values.

- If your collection's `_id` values are default generated [ObjectId](https://www.mongodb.com/docs/manual/reference/bson-types.md#std-label-objectid) values or ordered `int` or `long` values, `initialSync` achieves optimal performance. For other `_id` types, `initialSync` might take longer to complete because the values aren't stored in a predictable order. If the resume token is no longer on the oplog when `initialSync` completes, the processor enters a failed state to prevent data loss. To recover, restart the processor with `resumeFromCheckpoint=false`, which causes `initialSync` to run again. To learn more, see [Checkpoints.](https://www.mongodb.com/docs/atlas/atlas-stream-processing/architecture.md#std-label-atlas-sp-checkpoints)

- Atlas Stream Processing can insert new documents into a partition while `initialSync` copies it. A network failure that interrupts a query on a growing partition can prolong the collection-copy phase.

- `initialSync` might insert duplicate documents if it reads a change event during the collection-copy or catch-up phase. Atlas Stream Processing's at-least-once processing guarantee covers this behavior. Duplicates are more likely when your collection's `_id` values aren't `ObjectId` values or ordered `int` or `long` values, because the collection-copy phase runs longer and spans more change events.

## Failover Processors

- Atlas Stream Processing supports [failover processors](https://www.mongodb.com/docs/atlas/atlas-stream-processing/architecture.md#std-label-atlas-sp-architecture-failover) only for processors with the following configurations:

  - Atlas source and sink

  - Atlas source and [Apache Kafka](https://kafka.apache.org/) sink

- Atlas Stream Processing supports failover processors only for processors of [tier](https://www.mongodb.com/docs/atlas/atlas-stream-processing/architecture.md#std-label-atlas-sp-architecture-tiers) `SP10` or higher.

- A stream processing workspace can have up to two failover regions. Failover regions must use the same cloud provider as the stream processing workspace's default region. Atlas Stream Processing doesn't support cross-cloud failover.

- For processors configured with failover processors, only one processor can be active at any given time. You can edit only the active processor.