mongoDb queries getting slow down with logs

Hello everyone,
I am running a cron whose code is written in golang, and i am using mongoDb as database
There was 128GB Ram into my system in which DataBase is stored, and I am using different system for the code.
The cron is running with 17000 merchants parallely, each merchant having different database, which means there was 17000 Dbs into system,
Cron work is to send reminders based on the db data, and update data into db,
numbers of reminder can be variadic, let’s say for a merchant there may be 20 reminders to be sent, and for another merchant there was no reminder to be sent in a specific cron.
And the time of cron interval is 15 minutes.

Now I will tell you a scenario,
from 17000 merchants, there was around 6000 merchants in which reminders to be sent through a cron,
and in each merchant there was 5 reminders to be sent
while sending 1 reminder there was average 20-30 db queries(including Get, insert, update) run.

The issue is that, I have to save slow query logs, db logs for that db queries which makes db slow.
When the cron get starts, the db queries ran fast, but after sometime it gets slowed down.

Ram usage of mongoDb system goes to maximum of 85%
and free memory space will be around 15GB

After seeing the mongoDb logs, I have found that mongoDb stop executing queries for sometime, and after sometime it starts executing queries again.

and I found some abnormal things in the log after that mongoDb stop works for a while:-

“msg”:“Failed to gather storage statistics for slow operation”,“attr”:{“opId”:1533607,“error”:“lock acquire timeout”

Checkpoint has been running for 165 seconds and wrote: 35000 pages (1160 MB)

Can someone help me to find out why I am facing these issues?

The first looks like it fails to acquire a lock in order to gather storage statistics. I’m not sure if this slows down your cron or not.

This indicates that you are doing a lot of writes (my guess, not 100% sure ), and the checkpoint takes a longer time to finish. Check Internals - Checkpoints and Journaling - #3 by Alexandre_Araujo for more info.

As you have more merchants, i believe it will be difficult for your cron to catch up. (e.g. may run even longer than 15mins internal).

Hello @Kobe_W
Thanks for reply,
I understand what you are referring to and I am looking into it further, what I am not able to understand is that the mongodb hangs for few seconds and hang time increases eventually when this error occurs. In the hang time it does not perform any operations like find query etc. Also, if I disable the slow query logging then this should not happen but when I disabled the slow query loging then this error did not appear but the mongodb hang was happening. As suggested in productions notes, I have moved journal and logs to another drive but no improvement in the performance.