Hello everyone,
I am running a cron whose code is written in golang, and i am using mongoDb as database
There was 128GB Ram into my system in which DataBase is stored, and I am using different system for the code.
The cron is running with 17000 merchants parallely, each merchant having different database, which means there was 17000 Dbs into system,
Cron work is to send reminders based on the db data, and update data into db,
numbers of reminder can be variadic, let’s say for a merchant there may be 20 reminders to be sent, and for another merchant there was no reminder to be sent in a specific cron.
And the time of cron interval is 15 minutes.
Now I will tell you a scenario,
from 17000 merchants, there was around 6000 merchants in which reminders to be sent through a cron,
and in each merchant there was 5 reminders to be sent
while sending 1 reminder there was average 20-30 db queries(including Get, insert, update) run.
The issue is that, I have to save slow query logs, db logs for that db queries which makes db slow.
When the cron get starts, the db queries ran fast, but after sometime it gets slowed down.
Ram usage of mongoDb system goes to maximum of 85%
and free memory space will be around 15GB
After seeing the mongoDb logs, I have found that mongoDb stop executing queries for sometime, and after sometime it starts executing queries again.
and I found some abnormal things in the log after that mongoDb stop works for a while:-
“msg”:“Failed to gather storage statistics for slow operation”,“attr”:{“opId”:1533607,“error”:“lock acquire timeout”
Checkpoint has been running for 165 seconds and wrote: 35000 pages (1160 MB)
Can someone help me to find out why I am facing these issues?