Hello everyone,
I am working on a project, In which we are processing high amount of data.
We are using mongoDb as database into our project, I have setup replicas on my mongoDb, with 3 replica sets(1 as primary, 2 secondary)
and my system configuration is:-
RAM:- 128GB(In each system)
OS:- ubuntu 22.04
My problem is, whenever i have started the mongoDb server,
for sometime it was working fine, and after a few time, it becomes slow, it starts taking pause in between the processes,
And time of these pause are increasing rapidly,
e.g. when first time mongoDb stops, it will start in 1-5 seconds, and this time will increase rapidly,
after some time it will take 20s to start whenever it was stops, and this time will goes to 2-3 minutes during the end of the day.
And I am facing this mongodb stop issue in every 1-2 minutes.
While looking at the mongoDb logs, I have found some suspicious log that was logging continously, whenever the db stops working.
{“t”:{“$date”:“2023-03-16T12:39:32.469+00:00”},“s”:“D1”, “c”:“REPL”, “id”:21342, “ctx”:“conn9”,“msg”:“Waiting for an isMaster response from a topology change or until deadline”,“attr”:{“deadline”:{“$date”:“2023-03-16T12:39:42.469Z”},“currentTopologyVersionCounter”:4}}
{“t”:{“$date”:“2023-03-16T12:39:21.554+00:00”},“s”:“I”, “c”:“COMMAND”, “id”:51803, “ctx”:“conn33364”,“msg”:“Slow query”,“attr”:{“type”:“command”,“ns”:“admin.$cmd”,“command”:{“hello”:1,“helloOk”:true,“topologyVersion”:{“processId”:{“$oid”:“6412883cae50b6d202b8c8bc”},“counter”:4},“maxAwaitTimeMS”:10000,“$db”:“admin”,“$readPreference”:{“mode”:“primaryPreferred”}},“numYields”:0,“reslen”:788,“locks”:{},“protocol”:“op_msg”,“durationMillis”:0}}
Please help me out with this issue.
FYI, there was around 10k/s read operations and around 4-5k/s update and insert operations…