Allow the user to see live updates of the data

Hello,

I’m planning to build an application where users can work together concurrently. Therefore, each user will see all updates made by other users live, without significant delay.

Some database technologies, like Google’s Firebase Firestore, provide this functionality out of the box (e.g., onSnapshot).

I want to implement a similar mechanism and was hoping to do it with MongoDB. After some research, I discovered the existence of “change streams.” I plan to open one change stream for each collection that should be live updated for the user. Then, I would open a WebSocket for each user, where they can subscribe to the required data. The backend would connect to the existing change stream and only redirect the changed data that the current user subscribed to. Theoretically, this would work. However, I’m concerned about the number of change streams this solution could result in. Even though this solution would be an optimization because there isn’t a change stream for every active user, just for each collection, further research revealed limitations in the number of change streams: https://www.mongodb.com/docs/atlas/app-services/reference/service-limitations/#change-streams, along with the advice to “Avoid data models that require you to sync an unbounded number of collections.”

Now, I’m faced with a problem: my users are separated into groups, and I wanted to separate the data by user group (group is part of the collection path). This could potentially lead to an infinite number of change streams if the number of groups is not limited.

I know I could add an attribute in the document for the corresponding group, thus reducing the number of collections to one. However, I’m hesitant to merge the data from different groups because it involves very sensitive information.

Is there a way to circumvent this limitation? Or, would it be a better idea to track all data that the current users see and then start requests for that data at fixed intervals (perhaps every 5 seconds)? This method would simulate live update functionality but would generate a lot of traffic and introduce a small delay.

Does anyone have any ideas?

T