Using MongoDB Atlas on Lambda throws Could not connect to any servers in MongoDB Atlas cluster error

I am using node lambda which is a api server that accepts request from API gateway. The lambda connects to mongo db atlas to pull data from the db and then returns it in the response. For the most part it works fine with no issues but randomly it runs into mongo atlas connection errors:

    {
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "MongooseServerSelectionError: Could not connect to any servers in your MongoDB Atlas cluster. One common reason is that you're trying to access the database from an IP that isn't whitelisted. Make sure your current IP address is on your Atlas cluster's IP whitelist: https://docs.atlas.mongodb.com/security-whitelist/",
    "reason": {
        "errorType": "MongooseServerSelectionError",
        "errorMessage": "Could not connect to any servers in your MongoDB Atlas cluster. One common reason is that you're trying to access the database from an IP that isn't whitelisted. Make sure your current IP address is on your Atlas cluster's IP whitelist: https://docs.atlas.mongodb.com/security-whitelist/",
        "message": "Could not connect to any servers in your MongoDB Atlas cluster. One common reason is that you're trying to access the database from an IP that isn't whitelisted. Make sure your current IP address is on your Atlas cluster's IP whitelist: https://docs.atlas.mongodb.com/security-whitelist/",
        "reason": {
            "type": "ReplicaSetNoPrimary",
            "setName": null,
            "maxSetVersion": null,
            "maxElectionId": null,
            "servers": {},
            "stale": false,
            "compatible": true,
            "compatibilityError": null,
            "logicalSessionTimeoutMinutes": null,
            "heartbeatFrequencyMS": 10000,
            "localThresholdMS": 15,
            "commonWireVersion": null
        },
        "stack": [
            "MongooseServerSelectionError: Could not connect to any servers in your MongoDB Atlas cluster. One common reason is that you're trying to access the database from an IP that isn't whitelisted. Make sure your current IP address is on your Atlas cluster's IP whitelist: https://docs.atlas.mongodb.com/security-whitelist/",
            "    at NativeConnection.Connection.openUri (/var/task/node_modules/mongoose/lib/connection.js:846:32)",
            "    at /var/task/node_modules/mongoose/lib/index.js:351:10",
            "    at /var/task/node_modules/mongoose/lib/helpers/promiseOrCallback.js:31:5",
            "    at new Promise (<anonymous>)",
            "    at promiseOrCallback (/var/task/node_modules/mongoose/lib/helpers/promiseOrCallback.js:30:10)",
            "    at Mongoose._promiseOrCallback (/var/task/node_modules/mongoose/lib/index.js:1149:10)",
            "    at Mongoose.connect (/var/task/node_modules/mongoose/lib/index.js:350:20)",
            "    at Object.<anonymous> (/var/task/build/app.js:172:40)",
            "    at Module._compile (internal/modules/cjs/loader.js:999:30)",
            "    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1027:10)",
            "    at Module.load (internal/modules/cjs/loader.js:863:32)",
            "    at Function.Module._load (internal/modules/cjs/loader.js:708:14)",
            "    at Module.require (internal/modules/cjs/loader.js:887:19)",
            "    at require (internal/modules/cjs/helpers.js:74:18)",
            "    at _tryRequire (/var/runtime/UserFunction.js:75:12)",
            "    at _loadUserApp (/var/runtime/UserFunction.js:95:12)"
        ]
    },
    "promise": {},
    "stack": [
        "Runtime.UnhandledPromiseRejection: MongooseServerSelectionError: Could not connect to any servers in your MongoDB Atlas cluster. One common reason is that you're trying to access the database from an IP that isn't whitelisted. Make sure your current IP address is on your Atlas cluster's IP whitelist: https://docs.atlas.mongodb.com/security-whitelist/",
        "    at process.<anonymous> (/var/runtime/index.js:35:15)",
        "    at process.emit (events.js:314:20)",
        "    at process.EventEmitter.emit (domain.js:483:12)",
        "    at processPromiseRejections (internal/process/promises.js:209:33)",
        "    at processTicksAndRejections (internal/process/task_queues.js:98:32)"
    ]
}

My mongo DB atlas instance allows ip addresses from everywhere: 0.0.0.0.

This is how I am making my mongo connection:

import express from "express";
import serverless from "serverless-http";
import mongoose, { Error } from "mongoose";
import { APIGatewayProxyHandler } from 'aws-lambda';
import { AppSettings } from "./constants/constants";

class App {

    public app: express.Application;
    public mongoUrl: string = AppSettings.MONGODBURL;

    constructor() {
        this.setupDBConn();
    }
    async setupDBConn() {
        this.app = express();
    }

    private async mongoSetup() {
        await mongoClient;
        console.log('MONGO connection successfully made...');
    }
}
const mongoClient = mongoose.connect(AppSettings.MONGODBURL, { useNewUrlParser: true, useUnifiedTopology: true, useCreateIndex: true, useFindAndModify: false });
export const handler: APIGatewayProxyHandler = serverless(new App().app);

I am also using mongoose version 5.12.8

Hi @Shawn_Varughese and welcome in the MongoDB Community :muscle: !

Are you creating a new connection (==mongoClient) for each Lambda or your MongoDB Connections are shared across multiple (all) lambdas?

I give you a little hint, “yes” is a very bad answer :wink: !

Check out this doc where they explain how to set up the connection correctly:

Using a private endpoint would also be a plus (instead of 0.0.0.0/0).

Cheers,
Maxime.

Thanks for this and this is helpful I did go through this and we do have the connection outside the lambda function and the connection is shared in the code we have the mongoClient outside the App class and the serverless function just waits for the mongo connection and reuses it. Unfortunately we are still seeing this connection issue.

Any other ideas on why we might be seeing this?

I don’t understand why you would get this error if the connection was already established and functional and if your cluster is healthy on the other hand. This error message makes me think that the connection is being created at that moment.

If it’s really an occasional error, maybe this is happening when you have a maintenance on the cluster? Can you link this to an event in Atlas?
But in theory, it’s supposed to be transparent for the clients, unless something is misconfigured.

Interesting I am not sure that could be possible. I dont see any open alerts in Atlas is that what you were asking for? To give you more context our platform is microservices based so we have 8 microservices which are all separate lambda functions and each connect to atlas to read/write to the database.

You dont see this as a problem right?

Yes. Then it’s something else.

No, sounds like I would do exactly the same thing. Given that your connection pool is centralised and reused correctly by all the lambdas, maybe you could increase the size of the pool? If you have many lambdas running in parallel, I guess you need a connection pool large enough to accommodate all the queries.

So are you all 8 microservices should share the same connection pool? How would i be able to do that if they are all separate lambda functions

Or each lambda function shares its own connection pool and just does not create a new connection each invocation?

Ado proposed an implementation here based on a cache:

It’s fine if you use like 8 different pools as long as it’s a small number, it’s OK. What’s not OK is one connection to the cluster per lambda execution. That’s definitely a problem.

Your cluster can only support a limited number of connections. For an M10 it’s 1500 per node for example.

https://www.mongodb.com/docs/atlas/reference/atlas-limits/.

Keep an eye on the monitoring to see if you are getting close to that limit.

Cheers,
Maxime.

Thanks for this i was able to implementing the caching and it seems like it working well so far! Now after implementing caching i am facing a new error:
ERROR Unhandled Promise Rejection {
“errorType”: “Runtime.UnhandledPromiseRejection”,
“errorMessage”: “MongoNetworkTimeoutError: connection timed out”,
“reason”: {
“errorType”: “MongoNetworkTimeoutError”,
“errorMessage”: “connection timed out”,
“name”: “MongoNetworkTimeoutError”,
“stack”: [
“MongoNetworkTimeoutError: connection timed out”,
" at connectionFailureError (/var/task/node_modules/mongodb/lib/core/connection/connect.js:362:14)",
" at TLSSocket. (/var/task/node_modules/mongodb/lib/core/connection/connect.js:330:16)",
" at Object.onceWrapper (events.js:420:28)",
" at TLSSocket.emit (events.js:314:20)",
" at TLSSocket.EventEmitter.emit (domain.js:483:12)",
" at TLSSocket.Socket._onTimeout (net.js:483:8)",
" at listOnTimeout (internal/timers.js:554:17)",
" at processTimers (internal/timers.js:497:7)"
]
},
“promise”: {},
“stack”: [
“Runtime.UnhandledPromiseRejection: MongoNetworkTimeoutError: connection timed out”,
" at /var/runtime/index.js:35:15",
" at /opt/nodejs/node_modules/@lumigo/tracer/dist/tracer/tracer.js:265:37",
" at processTicksAndRejections (internal/process/task_queues.js:97:5)"
]
}

This doesn’t happen all the time just randomly I assume its because the cached db connection has timed out. Any advise here?

1 Like

@MaBeuLux88 any thoughts on this error i am getting now after implementing the cacheing?

1 Like

Hey @Shawn_Varughese,

So sorry for the terrible delay to answer. I had a baby on May 1st so I was a little distracted.

To be honest, I have no idea at all. If it’s happening “randomly”, is this happening maybe after your lambda wasn’t triggered for a long time?

What timeout have your defined? Did you try to increase them a bit so maybe this gives more opportunity for this connection to land?

Search for “timeout” in here and try to increase the relevant one maybe? I assume connectTimeoutMS here, no?

Cheers,
Maxime.