We currently have a Cron job that runs every night, and every 2-3 days the primary node fails and PyMongo throws a NotPrimaryError along with the error message - “The server is in quiesce mode and will shut down”. The script just retrieves data from the MongoDB and then inserts the necessary data into an SQL DB without inserting anything in Mongo.
The error last showed up on 01/21/2022 at 01:09:00 AM, and this is what the logs show before that:
{"t":{"$date":"2022-01-21T01:08:32.753+00:00"},"s":"I", "c":"COMMAND", "id":21581, "ctx":"conn32","msg":"Received replSetStepUp request"}
{"t":{"$date":"2022-01-21T01:08:32.753+00:00"},"s":"I", "c":"ELECTION", "id":4615661, "ctx":"conn32","msg":"Starting an election due to step up request"}
{"t":{"$date":"2022-01-21T01:08:32.754+00:00"},"s":"I", "c":"ELECTION", "id":21437, "ctx":"conn32","msg":"Skipping dry run and running for election","attr":{"newTerm":80}}
{"t":{"$date":"2022-01-21T01:08:32.754+00:00"},"s":"I", "c":"REPL", "id":6015303, "ctx":"conn32","msg":"Waiting for in-progress election to complete before finishing stepup"}
The number of concurrent connections before this is 131 which shouldn’t be a problem.
There are some spikes in CPU and memory usage when this happens, but there’s 30% CPU usage left and 6 GB RAM.
Any help is greatly appriciated. I haven’t worked with Mongo much, so if there’s any details that are important, let me know and I’ll add them to the post.