Change Streams limitations on MongoDB ATLAS

MongoDB Atlas
1

Hi all,

Currently on our deployments we are using change streams mongoDB feature to be notified of modification of some collections.
In few words our application is design as following:

  • A process called “P1” that post into database a list of tasks (into a collection named “C1”),
  • A second process called “P2” which subscribes to change into this collection C1 to deploy additional processes called “PT(x)” which will have to perform the task
  • Each “PT(x)” process opens 2 change streams to be notified of any modification on the task configuration and its status.
    The P1 process also opens a change stream on the “C1” collection to be notified of any modification on any task.
    All of these change streams are concerning the same collection, but as I can have lot of PT(x) processes in parallel, the total number of change streams can be important.

We plan to use MongoDB Atlas to deploy our application in public cloud. So, I wonder about the limitation described in the page https://docs.mongodb.com/realm/reference/service-limitations/#change-streams.

  1. Does this page describe a global limitation about Change Streams on MongoDB Atlas or is it specific to Realm?
  2. Will I encounter limitations on number of change streams using Atlas compared to local native deployment of Mongo?

Thanks in advance for your help
Best Regards

2

HI @Jacques , there is no limitation on the number of change streams if you open them directly on MongoDB Atlas (without Realm Sync or Triggers). However high number of change streams may lead to high resource consumption. In MongoDB 5.1 there is an optimization that allows for more efficient execution of the distinct change streams.

1 Like
3

This looks like something that you might want to consider kafka (or any message queue) to dispatch one single C1 change stream to many kafka consumer.

See https://docs.mongodb.com/kafka-connector/current/

4

If each change stream opens a connection, wouldn’t there be a limit based on the cluster tier here: https://www.mongodb.com/docs/atlas/reference/atlas-limits/

Or does each change stream not open a new connection?

5

what counts against the 5 limit in M0?

e.g. my client (webapp) can have many users simultaneously listening to a particular collection, does that mean i can only have 5 concurrent connections? what counts towards the 5 limit? is it a call to .watch counted as 1?

app.currentUser.db(DB).collection(COL).watch({ filter: { operationType: 'update' }})
1 Like
6

@Katya could you or another MongoDB team member provide some clarity here? I’d really appreciate it.

It’d also be great to understand if there are specific recommendations for scaling change streams in a heavy realtime application.

1 Like