Performance: Long networking time and huge difference between count and actual returning

Working with Data Node.js Frameworks
, , ,
1

I’m fairly new to MongoDB query optimizing. I have a query that runs quite slow when there are about 7000 not-too-complex documents from a single collection to be retrieved (about 5 to 6 MB of data). I’m using Node.js and Mongoose.
For analysis, I added some time measurements and also an additional “countDocuments” for comparison. However, the actual query needs to return all documents at once:

console.time('FullQuery')
await MyMongooseModel.find(findCondition)
console.timeEnd('FullQuery')

console.time('OnlyCount')
await MyMongooseModel.countDocuments(findCondition)
console.timeEnd('OnlyCount')

Result I get on my local Node.js server connecting to a local Mongo installation running in a Docker container:

FullQuery: 821.835ms
OnlyCount: 33.472ms

Connecting my local Node.js server to my remote Atlas instance:

FullQuery: 3.595s
OnlyCount: 42.642ms

My interpretation of that data:

  • Filtering is quite fast, 30-40ms is not a concern. Hence, adding additional indexes won’t help.
  • The time difference between the local and the remote Mongo instance is huge: more than 2.5 seconds. I think that’s far more than it should take to transfer 6 MB of data through a rather fast internet connection. Hence, I think there might be some configuration problem.

I now have the questions below:

  • Is my interpretation correct?
  • Is there anything I can do to improve performance in general even locally? unfortunately, limiting the number of query results and/or projections to limit the properties is not really an option in my use case.
  • What can I do to make the Atlas remote instance return the query faster?
  • I know this is not a Mongoose help forum, but asking anyway: Could the problem be in Mongoose and/or the underlying Node.js driver?
3

What is your query and document like? without this info, it’s difficult for us to give any more insights.

Anyway, i personally think 800+ms for 5MB data (running locally) is still slow. i suppose your query is not using an index.

4

Thanks for your answer. I cannot show any more details because of organisational secrets. Hower, the find condition is pretty simple, nothing special.

If a missing index is the problem I’m wondering why the count query is so fast and the other so slow. In my understanding, an index helps in the engine to filter out the documents you want to do something with. If the filtering part were to be the problem, the count query would be equally slow and there would be almost no difference between them. But that’s not the case. So, what am I overlooking?

5

Is the “6M” data size accurate? After you switch from local mongo server to mongo atlast (which is remote) the same code runs 2.5s longer. All those time are supposed to be spent on network transmission only. So either data size is big, or network is too slow?

6

Yes, six megabyte is correct. And my own network and internet connection is quite fast. But the connection between the Atlas server and their network access point is of course not under my control.

But I take that as confirmation that something weird is going on here. I’ll have to dig deeper.