Hi,
seeing high CPU on all cores on all primaries on a three shard cluster with 6 distributed mongos. All 5.0.6, latest nodejs (4.5.0). All replicasets are 3 voting 2 non voting and distributed globally on AWS with private network peeriing between them.
There are about 20 nodejs applications at each mongos, with a few changeStreams active on reasonably busy sharded collections (~300 updates/second).
We have tried various max/minPoolSize options on the applications with no effect. We have also tried setting a number of the mongod/mongos.conf setParameters with little or no change. Each primary mongod is seeing approximately 1200 concurrent connections, and a very high reconnection rate. Have searched everywhere without luck so hoping someone can assist. On mongos, there is a lot of logs like:
{"t":{"$date":"2022-04-12T08:58:49.368+00:00"},"s":"I", "c":"CONNPOOL", "id":22567, "ctx":"establishCursors cleanup","msg":"Ending idle connection because the pool meets constraints","attr":{"hostAndPort":"be-db-0.xxxx:27018","numOpenConns":41}}
and then reconnections to the same host pretty much straight away. Have set the maxPoolSize/minPoolSize to 100/10 respectively and there’s still a lot of reconnections.
Customer’s cluster has been rather unstable for the past 24 hours. Our workload is almost constant and continuous (RAM/CPU usually close to a flat line) and have checked that has not changed. Cannot identify any underlying AWS network issues and have even thought some noisy neighbor has appeared on the hardware but we’re clutching at straws.
Any assistance or suggestions very much appreciated.
Rob