How large can I make my collection?

Working with Data
#1

I have an app that involves processing tickets in temporal order. There are about 3,000 tickets added each day. To find the next ticket to process, I will do a query on timestamps (i.e. find all tickets with timestamps after the last processed timestamp).

So as tickets are processed, they stay in a collection that stores all the tickets. My question is, should I worry about a collection getting too large (it might eventually have like 100,000 documents)? Will this dramatically effect the performance of my queries? Or should I run something that automatically archives processed tickets that don’t need to be queried anymore?

#2

When I have a situation like that, I like to have 2 collections. The first with unprocessed documents and one for processed documents for historical data queries.

#3

Hello @Neil_Chowdhury, welcome to the forum!

If you are having 3000 tickets per day, in a year it will be more than a million documents. To query (and update) documents efficiently based on a field, index the field(s). Working with a million documents is not difficult, but as the time passes, the number of documents increase and it can tax on the system resources (like, index size and memory, as these are shared with other queries).

As @steevej has mentioned, maintain two collections. You can periodically (e.g., monthly or quarterly), move some of the already processed data to the history collection using a scheduled batch process.