I am trying to debug an odd problem we’re having - and to be honest I’m struggling. We’re performing a aggregate query and depending on the exact query (as it can be altered by the end-user), the client CPU goes to 100%, blocks the event queue while CPU load on the server is not noticeable.
So, lets except for the moment that the query in question is rubbish and really slow, why would the client go to 100% cpu and block the node event queue?
The code is something like:
async function RunReport(report) {
c.LogMe('pre RunReport');
var cursor = await c.db.collection("activity").aggregate(report);
c.LogMe('post RunReport');
var answer = await cursor.toArray();
c.LogMe('post cusor');
return answer;
}
The CPU load and blocking is at the line:
var answer = await cursor.toArray();
The results of the query is always small (as there is a limit in the aggregate) and if I’ve interpreted the docs correctly, I can’t see how this line could block even if it has to wait for ‘sometime’ for results to be returned?
So this query is basically built by the user, ie, adding various filters and it’s one of these filters that is causing a slow running query. There is an expectation from the user that some of these queries will take time to run and there are appropriate UI controls in place to show this.
I am happy to share the pipeline, however, what we’re really confused/concerned about is why the 100% load on the client side and the blocking of the nodejs event loop (while the DB is doing little).
It may be my misunderstanding, but with the line:
await cursor.toArray();
I assume that it effectively yields to the node event loop (allowing other tasks/queries to take place) until the cursor has processed all entries. Or are you saying, toArray() is effectually a blocking operation and we should use another approach to yield back to the event loop while the query is running on the DB?
Putting the question slightly differently, if I have a pipeline that takes 10 seconds to run, how do I run it without blocking the nodejs event loop for 10 seconds (with the nodejs process at 100% load)?
It looks like an operation that blocks your app cause you’re using await (this is how it should work)…
if you want to keep the flow that way and just not block the app use Promises instead of await - as in Promises the code doesn’t stop, it gets to the return statement once the data comes back from Mongo.
you need to take into account the size of the results because you’re keeping it inside memory - could cause indirectly a high CPU usage.
any way you can limit the results amount to ensure no abuse will be created by fetching data.
Thanks for the reply. So, I was under the assumption that toArray returns a promise, and the await, well, waits until the promise is resolved? It shouldn’t block the entire nodejs queue, ie, nothing else can run, no other incoming events are processed.
Yes, this may be something that I may misunderstanding, I was expected the whole pipeline to be run on the server side, not partly on the client? I will look into this more.
`async function RunReport(report) {
c.db.collection(“activity”).aggregate(report).then( (cursor) => {
do somthing…
});
}
in this case, the app isn’t blocking the code while fetching the data…
anyway I see that toArray() is a pretty heavy operation by itself: toArray()