Cannot reconnect to cluster during autoscaling

MongoDB Atlas
,
#1

We are finding that our application can maintain a connection to our Atlas cluster while it autoscales, but if we try to restart our application then it fails to connect. We get the error MongoServerSelectionError: The server is in quiesce mode and will shut down and MongoServerSelectionError: Server selection timed out after 30000 ms. Here is the error report we are getting.

We are having this occur quite often in production when we see spikes in traffic. In these cases, our Atlas cluster will scale up, as well as our Kubernetes pods. When the new pods are created, our application fails to connect.

According to the Cluster autoscaling docs: “Auto-scaling works on a rolling basis, meaning the process doesn’t incur any downtime.” but this does not seem to be the case for us.

#2

I noticed that the connection string suggested by Mongo Atlas is of the format:

mongodb+srv://<username>:<password>@<cluster name>-pri.<random code>.gcp.mongodb.net/<db name>

Is it possible that by appending -pri to the cluster name, we are only connecting to the primary node, which will then see downtime when its spun down? I noticed that we can remove the -pri and still connect to the DB.

#4

Hi @Francesco_Virga - Welcome to the community!

We are finding that our application can maintain a connection to our Atlas cluster while it autoscales, but if we try to restart our application then it fails to connect.

According to the Cluster autoscaling docs: “Auto-scaling works on a rolling basis, meaning the process doesn’t incur any downtime.” but this does not seem to be the case for us.

Can you confirm if only during auto-scale of the Atlas cluster, your application is still able to connect as per normal and that the disconnection and server selection errors only occur when both the Atlas cluster is auto-scaling and new pods are created.

After this clarification, could you also provide the following details / information:

  1. Connection string including any options being used (Please redact any sensitive information / credentials)
  2. Driver being used
  3. Driver version being used
  4. If you have tried connecting from a non-containerised environment during auto-scale to see if the disconnection errors occur as well

Is it possible that by appending -pri to the cluster name, we are only connecting to the primary node, which will then see downtime when its spun down? I noticed that we can remove the -pri and still connect to the DB.

I would recommend going over the details in this post here and let me know if you have any further questions regarding the additional -pri in the connection string. However, in short, the -pri that you see in the connection string is not associated with the primary node directly. You may find the information on the Private Connection Strings page useful as well.

Regards,
Jason

#6

Hi @Jason_Tran, thanks for getting back to me!

I revisited the issue and realized it also happens with running pods. I’ve been able to reproduce this locally by running our application and connecting to the DB. I created an issue in the Mongoose GitHub which outlines all the information and logs, see here.

  1. mongodb+srv://<user>:<pwd>@<cluster-name>-pri.<random-code>.gcp.mongodb.net/<db name>?retryWrites=true&w=majority got this directly from the Atlas UI.
  2. “mongoose” v6.1.6, which uses “mongodb” v4.2.2 (Nodejs)
  3. See 2.
  4. Yes, I’ve seen it occur even without autoscaling, simply running the server locally.