We have 600,000 documents in our atlas collection, soon to be over 2M. Once we created the proper indexes, an atlas search with" text" or “autocomplete” returns in less than a second. We need pagination so we tried mongoose library mongoose-paginate-v2, https://www.npmjs.com/package/mongoose-paginate-v2 it takes 15 seconds!
This can help you for pagination logic fast with $skip and $limit, just convert it to JS or NodeJS
def idlimit(page_size, last_id=None, query=None, collection=None):
"""Function returns `page_size` number of documents after last_id
and the new last_id.
"""
if query is None:
query = {}
if last_id is None:
# When it is first page
cursor = collection.find(query).limit(page_size)
else:
query['_id'] = {'$gt': last_id}
cursor = collection.find(query).limit(page_size)
# Get the data
data = [x for x in cursor]
if not data:
# No documents left
del cursor
return None, None
# Since documents are naturally ordered with _id, last document will
# have max id.
last_id = data[-1]['_id']
# Return data and last_id
del cursor
return data, last_id
Hey jonathan, I wound up just using atlas search with limit and skip. I implemented virtual scrolling on my vue client and it works really well. Thanks for the response