Developing a Facebook Chatbot with AWS Lambda and MongoDB Atlas

This post is part of our Road to re:Invent series. In the weeks leading up to AWS re:Invent in Las Vegas this November, we'll be posting about a number of topics related to running MongoDB in the public cloud.

Introduction

While microservices have been the hot trend over the past couple of years, serverless architectures have been gaining momentum by providing a new way to build scalable, responsive and cost effective applications. Serverless computing frees developers from the traditional cost and effort of building applications by automatically provisioning servers and storage, maintaining infrastructure, upgrading software, and only charging for consumed resources. More insight into serverless computing can be found in this whitepaper.

Amazon’s serverless computing platform, AWS Lambda, lets you run code without provisioning and running servers. MongoDB Atlas is Hosted MongoDB as a Service. MongoDB Atlas provides all the features of the database without the heavy operational lifting. Developers no longer need to worry about operational tasks such as provisioning, configuration, patching, upgrades, backups, and failure recovery. In addition, MongoDB Atlas offers elastic scalability, either by scaling up on a range of instance sizes or scaling out with automatic sharding, all with no application downtime. Together, AWS Lambda and MongoDB Atlas allow developers to spend more time developing code and less time managing the infrastructure.

Learn how to easily integrate an AWS Lambda Node.js function with a MongoDB database in this tutorial.

To demonstrate the power of serverless computing and managed database as a service, I’ll use this blog post to show you how to develop a Facebook chatbot that responds to weather requests and stores the message information in MongoDB Atlas.

Setting Up MongoDB Atlas

MongoDB Atlas provides multiple size options for instances. Within an instance class, there is also the ability to customize storage capacity and storage speed, as well as to use encrypted storage volumes. The number of virtual CPUs (vCPUs) – where a vCPU is a shared physical core or one hyperthread – increases as the instance class grows larger.

The M10, M20, and M30 instances are excellent for development and testing purposes, but for production it is recommended to use instances higher than M30. The base options for instances are:

• M0 - Variable RAM, 512 MB Storage
• M10 – 2 GB RAM, 10 GB Storage, 1 vCPU
• M20 – 4 GB RAM, 20 GB Storage, 2 vCPUs
• M30 – 8 GB RAM, 40 GB Storage, 2 vCPUs
• M40 – 16 GB RAM, 80 GB Storage, 4 vCPUs
• M50 – 32 GB RAM, 160 GB Storage, 8 vCPUs
• M60 – 64 GB RAM, 320 GB Storage, 16 vCPUs
• M100 – 160 GB RAM, 1000 GB Storage, 40 vCPUs

Register with MongoDB Atlas and use the intuitive user interface to select the instance size, region, and features you need.

Connecting MongoDB Atlas to AWS Lambda

Important note: VPC Peering is not available with MongoDB Atlas free tier (M0). If you use an M0 cluster, allow any IP to connect to your M0 cluster and switch directly to the Set up AWS Lambda section.

MongoDB Atlas enables VPC (Virtual Private Cloud) peering, which allows you to easily create a private networking connection between your application servers and backend database. Traffic is routed between the VPCs using private IP addresses. Instances in either VPC can communicate with each other as if they are within the same network. Note, VPC peering requires that both VPCs be in the same region. Below is an architecture diagram of how to connect MongoDB Atlas to AWS Lambda and route traffic to the Internet.

Figure 1: AWS Peering Architecture Architecture

For our example, a Network Address Translation (NAT) Gateway and Internet Gateway (IGW) is needed as the Lambda function will require internet access to query data from the Yahoo weather API. The Yahoo weather API will be used to query real-time weather data from the chatbot. The Lambda function we will create resides in the private subnet of our VPC. Because the subnet is private, the IP addresses assigned to the Lambda function cannot be used in public. To solve this issue, a NAT Gateway can be used to translate private IP addresses to public, and vice versa. An IGW is also needed to provide access to the internet.

The first step is to set up an Elastic IP address, which will be the static IP address of your Lambda functions to the outside world. Go to Services->VPC->Elastic IPs, and allocate a new Elastic IP address.

Next we will create a new VPC, which you will attach to your Lambda function.

Go to Services->VPC->Start VPC Wizard.

After clicking VPC wizard, select VPC with Public and Private Subnets.

Let’s configure our VPC. Give the VPC a name (e.g., “Chatbot App VPC”), select an IP CIDR block, choose an Availability Zone, and select the Elastic IP you created in the previous step. Note, the IP CIDR that you select for your VPC, must not overlap with the Atlas IP CIDR. Click Create VPC to set up your VPC. The AWS VPC wizard will automatically set up the NAT and IGW.

You should see the VPC you created in the VPC dashboard.

Go to the Subnets tab to see if your private and public subnets have been set up correctly.

Click on the Private Subnet and go to the Route Table tab in the lower window. You should see the NAT gateway set to 0.0.0.0/0, which means that messages sent to IPs outside of the private subnet will be routed to the NAT gateway.

Next, let's check the public subnet to see if it’s configured correctly. Select Public subnet and the Route Table tab in the lower window. You should see 0.0.0.0/0 connected to your IGW. The IGW will enable outside internet traffic to be routed to your Lambda functions.

Now, the final step is initiating a VPC peering connection between MongoDB Atlas and your Lambda VPC. Log in to MongoDB Atlas, and go to Clusters->Security->Peering->New Peering Connection.

After successfully initiating the peering connection, you will see the Status of the peering connection as Waiting for Approval.

Go back to AWS and select Services->VPC->Peering Connections. Select the VPC peering connection. You should see the connection request pending. Go to Actions and select Accept Request.

Once the request is accepted, you should see the connection status as active.

We will now verify that the routing is set up correctly. Go to the Route Table of the Private Subnet in the VPC you just set up. In this example, it is rtb-58911e3e. You will need to modify the Main Route Table (see Figure 1) to add the VPC Peering connection. This will allow traffic to be routed to MongoDB Atlas.

Go to the Routes tab and select Edit->Add another route. In the Destination field, add your Atlas CIDR block, which you can find in the Clusters->Security tab of the MongoDB Atlas web console:

Click in the Target field. A dropdown list will appear, where you should see the peering connection you just created. Select it and click Save.

Now that the VPC peering connection is established between the MongoDB Atlas and AWS Lambda VPCs, let’s set up our AWS Lambda function.

Set Up AWS Lambda

Now that our MongoDB Atlas cluster is connected to AWS Lambda, let’s develop our Lambda function. Go to Services->Lambda->Create Lambda Function. Select your runtime environment (here it’s Node.js 4.3), and select the hello-world starter function.

Select API Gateway in the box next to the Lambda symbol and click Next.

Create your API name, select dev as the deployment stage, and Open as the security. Then click Next.

In the next step, make these changes to the following fields:

• Name: Provide a name for your function – for example, lambda-messenger-chatbot
• Handler: Leave as is (index.handler)
• Role: Create a basic execution role and use it (or use an existing role that has permissions to execute Lambda functions)
• Timeout: Change to 10 seconds. This is not necessary but will give the Lambda function more time to spin up its container on initialization (if needed)
• VPC: Select the VPC you created in the previous step
• Subnet: Select the private subnet for the VPC (don’t worry about adding other subnets for now)
• Security Groups: the default security group is fine for now

Press Next, review and create your new Lambda function.

In the code editor of your Lambda function, paste the following code snippet and press the Save button:

'use strict';
<p>var VERIFY_TOKEN = "mongodb_atlas_token";</p>
<p>exports.handler = (event, context, callback) => {
var method = event.context["http-method"];
// process GET request
if(method === "GET"){
var queryParams = event.params.querystring;
var rVerifyToken = queryParams['hub.verify_token']
if (rVerifyToken === VERIFY_TOKEN) {
var challenge = queryParams['hub.challenge']
callback(null, parseInt(challenge))
}else{
callback(null, 'Error, wrong validation token');
}<br>
}
};

This is the piece of code we'll need later on to set up the Facebook webhook to our Lambda function.

Set Up AWS API Gateway

Next, we will need to set up the API gateway for our Lambda function. The API gateway will let you create, manage, and host a RESTful API to expose your Lambda functions to Facebook messenger. The API gateway acts as an abstraction layer to map application requests to the format your integration endpoint is expecting to receive. For our example, the endpoint will be our Lambda function.

Go to Services->API Gateway->[your Lambda function]->Resources->ANY.

Click on Integration Request. This will configure the API Gateway to properly integrate Facebook with your backend application (AWS Lambda). We will set the integration endpoint to lambda-messenger-bot, which is the name I chose for our Lambda function.

Uncheck Use Lambda Proxy Integration and navigate to the Body Mapping Templates section.

Select When there are no templates defined as the Request body passthrough option and add a new template called application/json. Don't select any value in the Generate template section, add the code below and press Save:

##  See http://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-mapping-template-reference.html
##  This template will pass through all parameters including path, querystring, header, stage variables, and context through to the integration endpoint via the body/payload
#set($allParams = $input.params())
{
"body-json" : $input.json('$'),
"params" : {
#foreach($type in $allParams.keySet())
    #set($params = $allParams.get($type))
"$type" : {
    #foreach($paramName in $params.keySet())
    "$paramName" : "$util.escapeJavaScript($params.get($paramName))"
        #if($foreach.hasNext),#end
    #end
}
    #if($foreach.hasNext),#end
#end
},
"stage-variables" : {
#foreach($key in $stageVariables.keySet())
"$key" : "$util.escapeJavaScript($stageVariables.get($key))"
    #if($foreach.hasNext),#end
#end
},
"context" : {
    "account-id" : "$context.identity.accountId",
    "api-id" : "$context.apiId",
    "api-key" : "$context.identity.apiKey",
    "authorizer-principal-id" : "$context.authorizer.principalId",
    "caller" : "$context.identity.caller",
    "cognito-authentication-provider" : "$context.identity.cognitoAuthenticationProvider",
    "cognito-authentication-type" : "$context.identity.cognitoAuthenticationType",
    "cognito-identity-id" : "$context.identity.cognitoIdentityId",
    "cognito-identity-pool-id" : "$context.identity.cognitoIdentityPoolId",
    "http-method" : "$context.httpMethod",
    "stage" : "$context.stage",
    "source-ip" : "$context.identity.sourceIp",
    "user" : "$context.identity.user",
    "user-agent" : "$context.identity.userAgent",
    "user-arn" : "$context.identity.userArn",
    "request-id" : "$context.requestId",
    "resource-id" : "$context.resourceId",
    "resource-path" : "$context.resourcePath"
    }
}

The mapping template will structure the Facebook response in the desired format specified by the application/json template. The Lambda function will then extract information from the response and return the required output to the chatbot user. For more information on AWS mapping templates, see the AWS documentation.

Go back to Services->API Gateway->[your Lambda function]->Resources->ANY and select Method Request. In the Settings section, make sure NONE is selected in the Authorization dropdown list. If not, change it NONE and press the small Update button.

Go back to the Actions button for your API gateway and select Deploy API to make your API gateway accessible by the internet. Your API gateway is ready to go.

Set Up Facebook Messenger

Facebook makes it possible to use Facebook Messenger as the user interface for your chatbot. For our chatbot example, we will use Messenger as the UI. To create a Facebook page and Facebook app, go to the Facebook App Getting Started Guide to set up your Facebook components.

To connect your Facebook App to AWS Lambda you will need to go back to your API gateway. Go to your Lambda function and find the API endpoint URL (obscured in the picture below).

Go back to your Facebook App page and in the Add Product page, click on the Get Started button next to the Messenger section. Scroll down and in the Webhooks section, press the Setup webhooks button. A New Page Subscription page window should pop up. Enter your API endpoint URL in the Callback URL text box and in the Verify Token text box, enter a token name that you will use in your Lambda verification code (e.g. mongodb_atlas_token). As the Facebook docs explain, your code should look for the Verify Token and respond with the challenge sent in the verification request. Last, select the messages and messaging_postbacks subscription fields.

Press the Verify and Save button to start the validation process. If everything went well, the Webhooks section should show up again and you should see a Complete confirmation in green:

In the Webhooks section, click on Select a Page to select a page you already created. If you don't have any page on Facebook yet, you will first need to create a Facebook page. Once you have selected an existing page and press the Subscribe button.

Scroll up and in the Token Generation section, select the same page you selected above to generate a page token.

The first time you want to complete that action, Facebook might pop up a consent page to request your approval to grant your Facebook application some necessary page-related permissions. Press the Continue as [your name] button and the OK button to approve these permissions. Facebook generates a page token which you should copy and paste into a separate document. We will need it when we complete the configuration of our Lambda function.

Connect Facebook Messenger UI to AWS Lambda Function

We will now connect the Facebook Messenger UI to AWS Lambda and begin sending weather queries through the chatbot. Below is the index.js code for our Lambda function. The index.js file will be packaged into a compressed archive file later on and loaded to our AWS Lambda function.

"use strict";
<p>var assert = require("assert");
var https = require("https");
var request = require("request");
var MongoClient = require("mongodb").MongoClient;</p>
<p>var facebookPageToken = process.env["PAGE_TOKEN"];
var VERIFY_TOKEN = "mongodb_atlas_token";
var mongoDbUri = process.env["MONGODB_ATLAS_CLUSTER_URI"];</p>
<p>let cachedDb = null;</p>
<p>exports.handler = (event, context, callback) => {
context.callbackWaitsForEmptyEventLoop = false;</p>
<p>var httpMethod;</p>
<p>if (event.context != undefined) {
httpMethod = event.context["http-method"];
} else {
//used to test with lambda-local
httpMethod = "PUT";
}</p>
<p>// process GET request (for Facebook validation)
if (httpMethod === "GET") {
console.log("In Get if loop");
var queryParams = event.params.querystring;
var rVerifyToken = queryParams["hub.verify_token"];
if (rVerifyToken === VERIFY_TOKEN) {
var challenge = queryParams["hub.challenge"];
callback(null, parseInt(challenge));
} else {
callback(null, "Error, wrong validation token");
}
} else {
// process POST request (Facebook chat messages)
var messageEntries = event["body-json"].entry;
console.log("message entries are " + JSON.stringify(messageEntries));
for (var entryIndex in messageEntries) {
var messageEntry = messageEntries[entryIndex].messaging;
for (var messageIndex in messageEntry) {
var messageEnvelope = messageEntry[messageIndex];
var sender = messageEnvelope.sender.id;
if (messageEnvelope.message && messageEnvelope.message.text) {
var onlyStoreinAtlas = false;
if (
messageEnvelope.message.is_echo &&
messageEnvelope.message.is_echo == true
) {
console.log("only store in Atlas");
onlyStoreinAtlas = true;
}
if (!onlyStoreinAtlas) {
var location = messageEnvelope.message.text;
var weatherEndpoint =
"https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20weather.forecast%20where%20woeid%20in%20(select%20woeid%20from%20geo.places(1)%20where%20text%3D%22" +
location +
"%22)&format=json&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys";
request(
{
url: weatherEndpoint,
json: true
},
function(error, response, body) {
try {
var condition = body.query.results.channel.item.condition;
var response =
"Today's temperature in " +
location +
" is " +
condition.temp +
". The weather is " +
condition.text +
".";
console.log(
"The response to send to Facebook is: " + response
);
sendTextMessage(sender, response);
storeInMongoDB(messageEnvelope, callback);
} catch (err) {
console.error(
"error while sending a text message or storing in MongoDB: ",
err
);
sendTextMessage(sender, "There was an error.");
}
}
);
} else {
storeInMongoDB(messageEnvelope, callback);
}
} else {
process.exit();
}
}
}
}
};</p>
<p>function sendTextMessage(senderFbId, text) {
var json = {
recipient: { id: senderFbId },
message: { text: text }
};
var body = JSON.stringify(json);
var path = "/v2.6/me/messages?access_token=" + facebookPageToken;
var options = {
host: "graph.facebook.com",
path: path,
method: "POST",
headers: { "Content-Type": "application/json" }
};
var callback = function(response) {
var str = "";
response.on("data", function(chunk) {
str += chunk;
});
response.on("end", function() {});
};
var req = https.request(options, callback);
req.on("error", function(e) {
console.log("problem with request: " + e);
});</p>
<p>req.write(body);
req.end();
}</p>
<p>function storeInMongoDB(messageEnvelope, callback) {
if (cachedDb && cachedDb.serverConfig.isConnected()) {
sendToAtlas(cachedDb, messageEnvelope, callback);
} else {
console.log(<code>=> connecting to database ${mongoDbUri});
MongoClient.connect(mongoDbUri, function(err, db) {
assert.equal(null, err);
cachedDb = db;
sendToAtlas(db, messageEnvelope, callback);
});
}
}

function sendToAtlas(db, message, callback) {
db.collection("records").insertOne({
facebook: {
messageEnvelope: message
}
}, function(err, result) {
if (err != null) {
console.error("an error occurred in sendToAtlas", err);
callback(null, JSON.stringify(err));
} else {
var message = Inserted a message into Atlas with id: ${result.insertedId};
console.log(message);
callback(null, message);
}
});
}

We are passing the MongoDB Atlas connection string (or URI) and Facebook page token as environment variables so we'll configure them in our Lambda function later on.

For now, clone this GitHub repository and open the README file to find the instructions to deploy and complete the configuration of your Lambda function.

Save your Lambda function and navigate to your Facebook Page chat window to verify that your function works as expected. Bring up the Messenger window and enter the name of a city of your choice (such as New York, Paris or Mumbai).

Store Message History in MongoDB Atlas

AWS Lambda functions are stateless; thus, if you require data persistence with your application you will need to store that data in a database. For our chatbot, we will save message information (text, senderID, recipientID) to MongoDB Atlas (if you look at the code carefully, you will notice that the response with the weather information comes back to the Lambda function and is also stored in MongoDB Atlas).

Before writing data to the database, we will first need to connect to MongoDB Atlas. Note that this code is already included in the index.js file.

function storeInMongoDB(messageEnvelope, callback) {
  if (cachedDb && cachedDb.serverConfig.isConnected()) {
    sendToAtlas(cachedDb, messageEnvelope, callback);
  } else {
    console.log(`=> connecting to database ${mongoDbUri}`);
    MongoClient.connect(mongoDbUri, function(err, db) {
      assert.equal(null, err);
      cachedDb = db;
      sendToAtlas(db, messageEnvelope, callback);
    });
  }
}

sendToAtlas will write chatbot message information to your MongoDB Atlas cluster.

function sendToAtlas(db, message, callback) {
  db.collection("records").insertOne({
    facebook: {
      messageEnvelope: message
    }
  }, function(err, result) {
    if (err != null) {
      console.error("an error occurred in sendToAtlas", err);
      callback(null, JSON.stringify(err));
    } else {
      var message = `Inserted a message into Atlas with id: ${result.insertedId}`;
      console.log(message);
      callback(null, message);
    }
  });
}

Note that the storeInMongoDB and sendToAtlas methods implement MongoDB's recommended performance optimizations for AWS Lambda and MongoDB Atlas, including not closing the database connection so that it can be reused in subsequent calls to the Lambda function.

The Lambda input contains the message text, timestamp, senderID and recipientID, all of which will be written to your MongoDB Atlas cluster. Here is a sample document as stored in MongoDB:

{
  "_id": ObjectId("58124a83c976d50001f5faaa"),
  "facebook": {
    "message": {
      "sender": {
        "id": "1158763944211613"
      },
      "recipient": {
        "id": "129293977535005"
      },
      "timestamp": 1477593723519,
      "message": {
        "mid": "mid.1477593723519:81a0d4ea34",
        "seq": 420,
        "text": "San Francisco"
      }
    }
  }
}

If you'd like to see the documents as they are stored in your MongoDB Atlas database, download MongoDB Compass, connect to your Atlas cluster and visualize the documents in your fbchats collection:

Note that we're storing both the message as typed by the user, as well as the response sent back by our Lambda function (which comes back to the Lambda function as noted above).

Using MongoDB Atlas with other AWS Services

In this blog, we demonstrated how to build a Facebook chatbot, using MongoDB Atlas and AWS Lambda. MongoDB Atlas can also be used as the persistent data store with many other AWS services, such as Elastic Beanstalk and Kinesis. To learn more about developing an application with AWS Elastic Beanstalk and MongoDB Atlas, read Develop & Deploy a Node.js App to AWS Elastic Beanstalk & MongoDB Atlas.

To learn how to orchestrate Lambda functions and build serverless workflows, read Integrating MongoDB Atlas, Twilio, and AWS Simple Email Service with AWS Step Functions.

For information on developing an application with AWS Kinesis and MongoDB Atlas, read Processing Data Streams with Amazon Kinesis and MongoDB Atlas.

To learn how to use your favorite language or framework with MongoDB Atlas, read Using MongoDB Atlas From Your Favorite Language or Framework.

About the Author - Raphael Londner

Raphael Londner is a Principal Developer Advocate at MongoDB, focused on cloud technologies such as Amazon Web Services, Microsoft Azure and Google Cloud Engine. Previously he was a developer advocate at Okta as well as a startup entrepreneur in the identity management space. You can follow him on Twitter at @rlondner.