Building a notification system - How will it handle millions of documents?

I want to build a user notification system for my website. Is the right way to go about this to have one “notifications” collection where all notifications of all users are in?

I would prefer to never delete an old notification. This means that over time there will be millions, maybe even billions of documents in this notifications collection. Will this cause any problems when it comes to retrieving these notifications? Will I be able to retrieve the latest notifications for a specific user quickly enough (within seconds)?

Hi @Florian_Walther,

Will this cause any problems when it comes to retrieving these notifications? Will I be able to retrieve the latest notifications for a specific user quickly enough (within seconds)?

Regarding the “specific user”, you could create an index for the associated field containing the user’s unique identifier (and any additional specifications e.g. date, notification messages, etc) to assist with performance. However, this is based off my interpretation of what fields the document(s) in the notification collection have.

In terms of the latest notification, I interpret this as a newly-inserted notification document. I believe when the document in question is still within your working set, it should be fast to retrieve, the total size of the collection notwithstanding. However if your goal is to never delete any document, you might be interested in taking a look at Atlas Online Archive which will automatically archive old documents, while still having the capability of querying them.

In saying so, for optimal performance you will essentially want your working set and commonly used indexes to fit in memory to prevent minimal reads from disk.

Regards,
Jason

3 Likes

Thank you for the tips! It seems like I can implement all these techniques (archiving, optimized indexes) in hindsight, right? So I might just go with a single collection now and implement these once the queries start to become slow.

It seems that by archiving old notification documents, the total amount of old notifications shouldn’t play a role anymore (for performance). It makes a lot of sense.

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.