Methods to make the collscan of 100k documents faster

Working with Data
1

The database has around 100k collections. Each document has an id field and multiple array fields that itself have multiple values.

The usecase is to get the id field and no. of elements in these arrays of all the collections and return the data.

Right now what i am doing is using a project stage and size operator to achieve the above. However it takes a lot of time. What can be done to reduce the time taken for this usecase

2

Hi @Prati123

The array type does not have a length property so each time the $size is scanning the array.

To optimise for this you can use a computed pattern. Whenever the array is manipulated update another field(arrayCount) with the updated count.

With a compound index on id and arrayCount querying and projecting this combination will be a covered query which will be fast.

https://www.mongodb.com/blog/post/building-with-patterns-the-computed-pattern
https://www.mongodb.com/docs/manual/core/query-optimization/#covered-query

1 Like
3

Hey @chris Thank you for your response. I here was trying to optimse my query, but your advice definetly provided a new way to approach this issue. I will be implementing a server side functions and checking the performance again