Write Operations Fail When Primary Replica is Down with Node.js

Working with Data Node.js Frameworks
,
1

Setup

I am running a MongoDB replica set with 3 replicas and 1 arbiter:

rs.initiate({
    _id: 'rs0',
    members: [
       { _id: 0, host: 'psmdb1:27017' },
       { _id: 1, host: 'psmdb2:27017' },
       { _id: 2, host: 'psmdb3:27017' },
       { _id: 3, host: 'psmdb-arbiter:27017', arbiterOnly: true }
    ]
}, {
    settings: {
        electionTimeoutMillis: 1000,
        heartbeatIntervalMillis: 1000
    }
});

I connect to MongoDB using Node.js with the following configuration:

const url = 'mongodb://psmdb1:27017,psmdb2:27017,psmdb3:27017/?replicaSet=rs0&retryWrites=true&w=majority&connectTimeoutMS=10000&socketTimeoutMS=10000';

const options = {
  retryWrites: true,
  retryReads: true,
  socketTimeoutMS: 30000,
  connectTimeoutMS: 30000,
  serverSelectionTimeoutMS: 30000,
  heartbeatFrequencyMS: 1000
};

// Create a new MongoClient
const client = new MongoClient(url, options);

Problem

During a write operation, if the primary replica goes down, MongoDB does not retry the operation after a new primary is elected. Instead, it throws this error:

MongoBulkWriteError: connection 1 to 172.18.0.5:27017 closed

Even with retryWrites=true, it does not wait for the election to complete and loses data if a write operation is in progress.

Question

How can I ensure that MongoDB waits for the new primary and automatically retries writes without throwing errors or losing data? Is there any additional setting in MongoDB Node.js driver or replica set config to handle this?

Thanks in advance!

2

It looks like your issue is caused by how the MongoDB driver handles elections. Even with retryWrites=true, writes that fail during an election are not retried automatically.

  1. Increase serverSelectionTimeoutMS

    • Ensure the driver waits for a new primary instead of failing immediately:
const options = {   
  retryWrites: true,   
  retryReads: true,   
  serverSelectionTimeoutMS: 5000, // Wait 5 sec for primary   socketTimeoutMS: 30000,   
  connectTimeoutMS: 30000,   
  heartbeatFrequencyMS: 1000 
};

  1. Implement a Retry Mechanism in Code

    • MongoDB doesn’t automatically retry writes during an election, so you need to handle retries manually:
async function safeWriteOperation() {
  let retries = 3;
  while (retries > 0) {
    try {
      await client.db("test").collection("data").insertOne({ name: "MongoDB" });
      console.log("Write successful");
      break;
    } catch (err) {
      if (err.message.includes("closed") || err.message.includes("NotPrimary")) {
        console.log("Write failed, retrying...", retries);
        retries--;
        await new Promise(res => setTimeout(res, 2000));
      } else {
        throw err;
      }
    }
  }
}

  1. Ensure wtimeoutMS is Set for Write Concern

    • This ensures that writes wait for a new primary to be elected:
const url = 'mongodb://psmdb1:27017,psmdb2:27017,psmdb3:27017/?replicaSet=rs0&retryWrites=true&w=majority&wtimeoutMS=5000';

Some related docs…

3

Thank you @Michael_Lynn

For the last days I tried different approaches to make mongo driver reconnect to the replica set on the fly (timeouts, different connection strings, etc). Unfortunately, I did not find a reliable solution. I still meet different issues like MongoBulkWriteError.

So it seems that the best solution is to grasefully process mongo operations and retry them manually if they fail.

Thanks for your help!

Best regards,
Dmitry Egorov