Getting MongoServerSelectionError: ECONNREFUSED and Not primary

Hello,

I am getting the below error logs after the MongoDB election occur.

(node:9837) UnhandledPromiseRejectionWarning: MongoServerSelectionError: connect ECONNREFUSED 192.168.50.50:27019
    at Timeout._onTimeout
(node:9837) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 52)

192.168.50.50 It was primary node before MongoDB election is occurred, After MongoDB election the NodeJS application still trying to connect with old primary node(192.168.50.50) which was shut down due to some reason. And after sometime when 192.168.50.50 node is running up, It will become secondary MongoDB node and NodeJS application trying to connect with same IP (192.168.50.50 currently this is secondary node). Hence now application getting the below error.

(node:9837) UnhandledPromiseRejectionWarning: MongoServerSelectionError: not primary
    at Timeout._onTimeout
(node:9837) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode)

Note: IT IS RANDOM ISSUE AND GETTING ON RANDOM NODEJS SERVER

Why MongoDB node driver connect with old mongo instance?
Why it behave randomly?

Application structure:

  1. I have socket.io cluster with 9 NodeJS server deployed in kubernetes environment
  2. MongoDB replica-set cluster with 3 node and deployed on bare metal.
  3. Mongo version 4.4.8
    4 Node version 12.16.1
  4. Node driver version for MongoDB: 4.9.1
async function runAdapter() {
                            try {
                                const {
                                    MongoClient
                                } = require('mongodb');
                                const client = new MongoClient(common.appConfigObj.mongoConfig.url, common.MongoClientOption);
                                await client.connect();
                                const db = await client.db(common.appConfigObj.mongoConfig.dbName);
                                db.createCollection(common.appConfigObj.mongoConfig.cappedCollection, {
                                    capped: true,
                                    size: common.appConfigObj.mongoConfig.size ? common.appConfigObj.mongoConfig.size : 100000,
                                    max: common.appConfigObj.mongoConfig.max ? common.appConfigObj.mongoConfig.max : 5000,
                                }, (err, data) => {
                                    if (err) {
                                        logger.info(`In HapiServer : hapiServerFun : mongo : error while creating the cappped collection : ${err} `);
                                        if (err.message == `Collection already exists. NS: ${common.appConfigObj.mongoConfig.dbName}.${common.appConfigObj.mongoConfig.cappedCollection}`)
                                            connectAdapter(db.collection(common.appConfigObj.mongoConfig.cappedCollection))
                                        else
                                            runAdapter();
                                    } else if (data) {
                                        logger.info(`In HapiServer : hapiServerFun : mongo : capped collection successfully created `);
                                        connectAdapter(db.collection(common.appConfigObj.mongoConfig.cappedCollection))
                                    }
                                })
                                const connectAdapter = (coll) => {
                                    socketIO.adapter(createAdapter(coll));
                                    coll.isCapped().then((data) => {
                                        logger.info(`In HapiServer : hapiServerFun : mongo : cappedCollection : data : ${data}`);
                                    }).catch((err) => {
                                        logger.error(`In HapiServer : hapiServerFun : mongo : cappedCollection : error : ${err}`);
                                    })
                                    logger.info(`In HapiServer : hapiServerFun : mongo : cappedCollection : `);
                                    logger.info(`In HapiServer : hapiServerFun : mongo : mongo connection done`);
                                    console.log(`In HapiServer : hapiServerFun : mongo : mongo connection done`)
                                    return;
                                }
                            } catch (err) {
                                console.log(`In HapiServer : hapiServerFun : mongo connection Exception : : ${err}`)
                                logger.info(`In HapiServer : hapiServerFun : mongo connection Exception : ${err}`);
                                logger.error(`In HapiServer : hapiServerFun : mongo connection Exception : ${err}`);
                                return runAdapter();
                            }
                        }
                        runAdapter();

Can anyone help to find out the issue. Please let me know if you want anything else.

What does the common.appConfigObj.mongoConfig.url and common.MongoClientOption look like?

Redacting the sensitive details.

common.appConfigObj.mongoConfig.url → It contains the mongodb url (mongodb://username:password@192.168.50.50:27019,192.168.50.51:27019,192.168.50.52:27020/dbName?replicaSet=mongo-cluster)

common.MongoClientOption —> It contains the option object which is passed while connecting to mongo.

{
    appname: 'NodeProject',
    connectTimeoutMS: 20000,
}

Please see the below sample code.

async function runAdapter() {
    try {
        const {
            MongoClient
        } = require('mongodb');
        const client = new MongoClient('mongodb://username:password@192.168.50.50:27019,192.168.50.51:27019,192.168.50.52:27020/dbName?replicaSet=mongo-cluster', {
            appname: 'NodeProject',
            connectTimeoutMS: 20000,
        });
        await client.connect();
        const db = await client.db('dbName');
        db.createCollection('socketClusterData', {
            capped: true,
            size: 100000,
            max: 5000,
        }, (err, data) => {
            if (err) {
                logger.info(`In HapiServer : hapiServerFun : mongo : error while creating the cappped collection : ${err} `);
                    runAdapter();
            } else if (data) {
                logger.info(`In HapiServer : hapiServerFun : mongo : capped collection successfully created `);
                connectAdapter(db.collection('socketClusterData'))
            }
        })
        const connectAdapter = (coll) => {
            socketIO.adapter(createAdapter(coll));
            coll.isCapped().then((data) => {
                logger.info(`In HapiServer : hapiServerFun : mongo : cappedCollection : data : ${data}`);
            }).catch((err) => {
                logger.error(`In HapiServer : hapiServerFun : mongo : cappedCollection : error : ${err}`);
            })
            logger.info(`In HapiServer : hapiServerFun : mongo : cappedCollection : `);
            logger.info(`In HapiServer : hapiServerFun : mongo : mongo connection done`);
            console.log(`In HapiServer : hapiServerFun : mongo : mongo connection done`)
            return;
        }
    } catch (err) {
        console.log(`In HapiServer : hapiServerFun : mongo connection Exception : : ${err}`)
        logger.info(`In HapiServer : hapiServerFun : mongo connection Exception : ${err}`);
        logger.error(`In HapiServer : hapiServerFun : mongo connection Exception : ${err}`);
        return runAdapter();
    }
}
runAdapter();

Variable details:

socketIO: It is a socket server instance. (socket.io library of nodejs)
createAdapter: It is a adapter function to make socket.io cluster (@socket.io/mongo-adapter library of nodejs)

Let me know if you need anything else.

Is each mongo ip:port accessible by the kubenetes nodes and/or application containers?

It almost seems like the app can only connect to 192.168.50.50:27019

Is each mongo ip:port accessible by the kubenetes nodes and/or application containers?
Ans: Yes

It almost seems like the app can only connect to 192.168.50.50:27019
Reply:
No, If we restart the node application then application connect to primary node only. Whenever this scenario is occur, we restarting the node application and then it working fine.

That is great information @PRAVIN_DASARI

Do you know how long it take for your new primary to be elected on your cluster? You could try raising your serverSelectionTimeoutMS for the driver, but with the default at 30s I’d be interested in why an election is taking so long.

Do you know how long it take for your new primary to be elected on your cluster?
Ans: It take around 10s to elect new primary.

I have raise the time for server selection timeout. (serverSelectionTimeoutMS: 60000) But I’m still facing the same issue on a random node server instance.

Find the below error log

(node:15687) UnhandledPromiseRejectionWarning: MongoServerSelectionError: Server selection timed out after 60000 ms
    at Timeout._onTimeout (/usr/local/share/packages/node-v12.16.1-linux-x64/lib/node_modules/mongodb/lib/sdam/topology.js:293:38)
    at listOnTimeout (internal/timers.js:549:17)
    at processTimers (internal/timers.js:492:7)
(node:15687) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 93)

Why driver is unable to find the new primary mongo node? And why it behave randomly?

Hello @chris

Do you know any other solution?

As next step I would recommend updating MongoDB to the latest available 4.4 release 4.4.8 is not recommend for production use.

Likewise updating the node driver to the latest available release.

And seeing if the problem persists.

The last 2-3 years have been in a cycle faster release/patch cycle. MongoDB has seen 4.x, 5x and now 6x, as nodejs driver also saw more than 10 of them from 3x family to 4.x.

For this reason, things can go sideways anytime unexpectedly. the problem you are having might be a one-off issue on that specific driver version or specific database version (and maybe fixed in next patch version). It is easier to try a few different nodejs versions (both nodejs itself and mongodb driver) compared to replacing the database version. try up(down)grading to driver v4.8 and 4.10 for example.

PS: is there any specific reason you do not use the 27017 port on your replica members?

Hello @chris

I’ll update MongoDB and will check the same scenario. The latest node driver version required node v14.x but I have some internal dependencies which are not supported node v14.x. I’ll figure out how to handle internal dependency with node v14.x and get back to you.

Hello @Yilmaz_Durmaz,

Due to some security reasons, we didn’t use the default port.

was it always like this or started after some upgrades? it is possible you have a missing configuration from your replica set members or their host machines, such as firewall settings not identical to the primary.

this one is weird! Nodejs v16 and v18 (current LTS) are in use for quite a long time and the latest driver should be working with v18. if your app does not use any v14-specific functions, try v18 instead.

@Yilmaz_Durmaz

I am moving to the latest driver and it required node v14.x+. and I am updating it to the current LTS(node v18.x).

Will let you know once done all the things.