Connecting to MongoDB Atlas from Google Cloud Functions

MongoDB Atlas
,
1

I am connecting to MongoDB Atlas with Google Cloud Functions (Serverless). I see that there is a high number of active connections on the server that is hosting my replica while I am the only user currently running the functions. (My app is still in development)
I found the following article that solves the issue for AWS Lambda functions (https://docs.atlas.mongodb.com/best-practices-connecting-to-aws-lambda) But I am not able to find the Google Cloud equivalent. does anyone knows how can I reuse my connection between Cloud Function instances?

2

Hi @Atef_Sawaed,

Welcome to MongoDB community.

I have not used google functions against Atlas but I wonder if global variables could help here the same way as the concept of handlers work in AWS lambda:

Try to define the MongoDB client as a global variable.

Thanks
Pavel

3

Hi Pavel,

Yes, global variables are supported in GC Functions and I’ve already used them. Here is my implementation:

var client;

const getClient = async () => {
	if (client && client.isConnected()) {
		console.log("MONGODB CLIENT ALREADY CONNECTED!");
	} else
		try {
			client = await MongoClient.connect(url, {
				useNewUrlParser: true,
				useUnifiedTopology: true,
			});
			console.log("MONGODB CLIENT RECONNECTED!");
		} catch (e) {
			throw e;
		}

	return client;
};

exports.getPlacesFromDB = functions.https.onCall((data, context) => {
    var limitOption = 10;
    var query = {};
    var sortOption = {};
    return getClient().then((client) => {
        const db = client.db("database");
        const collection = db.collection("collection");
        
        var _cursor = collection.find(query).skip(skips).limit(limitOption).sort(sortOption);
        return _cursor.map(({ _id, ...d }) => ({ _id: _id.toString(), ...d })).toArray().then((result) => {
            // Returning the message to the client.
            return { text: 'Retrieved data!', data: result };
          })
          .catch((error) => {
              functions.logger.log("error:", error.message);
              throw new functions.https.HttpsError('cannot get data!', error.message, error);
            });
        }).catch((error) => {
          functions.logger.log("error:", error.message);
          throw new functions.https.HttpsError('no connection', error.message, error);
        });
});

In the AWS article I shared, they are suggesting to disable callbackWaitsForEmptyEventLoop in the Context object of the Lambda function. I am trying to see how can this be done in GCP?

Thanks!

4

@Atef_Sawaed,

Maybe background functions? :man_shrugging:t2:

5

@Pavel_Duchovny,

How can background functions help me here? Can you elaborate?

I Appreciate your support and quick responses!

6

@Atef_Sawaed,

It was just a guess to allow usage of callback rather than freeze.

With global variable do you still see unexpected amount of connections?

Thanks
Pavel

7

Yes. I see ~40 active connections in my cluster while there is 1-2 daily active users.

8

@Atef_Sawaed,

Usually there are different parties connecting to the Atlas nodes like the other nodes and our agents.

It is possible that those together with connection pools from the Google functions endup in 40 connections.

I will suggest you to contact our support for a more specific breakdown.

thanks,
Pavel

9

@Atef_Sawaed Any updates on this ? Could you find a proper solution in the end ?
Thanks in advance !

10

@Atef_Sawaed @Benjamin_Bouvier Any updates on this?
I’m having the same problem with a lot of Exit with code 16 errors in my cloud functions due to this.
Do you have a pattern to properly manage connections in Google Cloud Functions?

Thanks in advance

11

Hi @Atef_Sawaed and @Axel_Vaindal

I didn’t find any official best practices.

But found some interesting articles

Hope you can use it

Ty

12

Not sure this is the root cause, but GCP Functions different from AWS Lambda in a way that GCP Functions allows one instance to handle multiple requests concurrently. A proper resource guard is therefore required to avoid creating multiple clients when concurrent requests arrive. @Atef_Sawaed’s example can be modified to something like below (WARNING: I haven’t tested the code):

const getClient = async () => {
	if (client && client.isConnected()) {
		console.log("MONGODB CLIENT ALREADY CONNECTED!");
	} else if (client instanceof Promise) {
		client = await client;
		console.log("MONGODB CLIENT RECONNECTED!");
	} else {
		// If client.isConnected() return false and arrived here, do we need explicit disconnect and release resource?
		client = MongoClient.connect(url, {
			useNewUrlParser: true,
			useUnifiedTopology: true,
		});
	}
	return client;
};