Data Enrichment with MongoDB Stitch
Objectives
Here is what we are going to achieve in this tutorial:
Firstly, we are going to write a document to MongoDB using
MongoDB Stitch
.
The result in our MongoDB collection will look like this:
{
"_id": ObjectId("5bb27712dced5f37bebf388c"),
"Title":"Guardians of the Galaxy"
}
Secondly, a trigger will catch this new insertion and start a function.
Lastly, this function will call
the OMDB external API
with the given movie title, fetch data about that movie, and finally enrich our MongoDB document with the data we gathered from this API.
This is the final result we expect in our MongoDB collection:
{
"_id": ObjectId("5bb27712dced5f37bebf388c"),
"Title":"Guardians of the Galaxy",
"Year":"2014",
"Rated":"PG-13",
"Released":"01 Aug 2014",
"Runtime":"121 min",
"Genre":"Action, Adventure, Comedy",
"Director":"James Gunn",
"Writer":"James Gunn, Nicole Perlman, Dan Abnett (based on the Marvel comics by), Andy Lanning (based on the Marvel comics by), Bill Mantlo (character created by: Rocket Raccoon), Keith Giffen (character created by: Rocket Raccoon), Jim Starlin (characters created by: Drax the Destroyer, Gamora & Thanos), Steve Englehart (character created by: Star-Lord), Steve Gan (character created by: Star-Lord), Steve Gerber (character created by: Howard the Duck), Val Mayerik (character created by: Howard the Duck)",
"Actors":"Chris Pratt, Zoe Saldana, Dave Bautista, Vin Diesel",
"Plot":"A group of intergalactic criminals are forced to work together to stop a fanatical warrior from taking control of the universe.",
"Language":"English",
"Country":"USA",
"Awards":"Nominated for 2 Oscars. Another 52 wins & 99 nominations.",
"Poster":"https://m.media-amazon.com/images/M/MV5BMTAwMjU5OTgxNjZeQTJeQWpwZ15BbWU4MDUxNDYxODEx._V1_SX300.jpg",
"Ratings":[
{
"Source":"Internet Movie Database",
"Value":"8.1/10"
},
{
"Source":"Rotten Tomatoes",
"Value":"91%"
},
{
"Source":"Metacritic",
"Value":"76/100"
}
],
"Metascore":"76",
"imdbRating":"8.1",
"imdbVotes":"871,949",
"imdbID":"tt2015381",
"Type":"movie",
"DVD":"09 Dec 2014",
"BoxOffice":"$270,592,504",
"Production":"Walt Disney Pictures",
"Website":"http://marvel.com/guardians",
"Response":"True"
}
Prerequisites
So first of all, if you want to try this at home, it is very easy. The only requirement here is to create a free
MongoDB Atlas
cluster.
This video
will show you the steps.
MongoDB Stitch
is our serverless platform, built by MongoDB on top of MongoDB Atlas. Once our MongoDB Atlas cluster is ready to use, link a MongoDB Stitch application to it:
Click on the left panel on “Stitch Apps”,
Then click on “Create New Application”,
Pick the name you want for your application,
Link it to your free MongoDB Atlas cluster.
Actions
To be able to send a document to MongoDB, we are going to use an HTTP POST service.
On the left panel, click on “Services”,
Then click on “Add a Service”,
Choose a service name “IMDB”,
Note: “IMDB” will be reuse later in the function code. If you choose another name, please make sure to update the code accordingly.
Click on “Add Service”,
Click on “Add Incoming Webhook”,
And copy the screenshot below.
When this is done, hit the “Save” button and you will be in the “Function Editor” screen.
Enter the following code:
exports = function(payload, response) {
const mongodb = context.services.get("mongodb-atlas");
const movies = mongodb.db("stitch").collection("movies");
var body = EJSON.parse(payload.body.text());
movies.insertOne(body)
.then(result => {
response.setStatusCode(201);
});
};
Click the “Save” button again.
Now that our Service is ready, we can test it!
Go to the “Settings” and you will find your Webhook URL.
You can now send an HTTP POST request like this to MongoDB Stitch:
curl -H "Content-Type: application/json" -d '{"Title":"Guardians of the Galaxy"}' https://webhooks.mongodb-stitch.com/api/client/v2.0/app/stitchtapp-abcde/service/IMDB/incoming_webhook/post_movie_title?secret=test
Note: I used a curl command but feel free to use Postman or whatever you are used to.
We can check it worked by having a look at the content of the “stitch.movies” collection in our MongoDB Atlas Cluster:
Now that we can insert a new document into MongoDB Atlas using Stitch, we are going to create the trigger.
On the left panel click on “Triggers”,
Then click on “Add a Database Trigger”,
And do as follow.
This is the code you will need to create the new function:
exports = function(changeEvent) {
var docId = changeEvent.documentKey._id;
var title = encodeURIComponent(changeEvent.fullDocument.Title.trim());
var movies = context.services.get("mongodb-atlas").db("stitch").collection("movies");
var imdb_url = "http://www.omdbapi.com/?apikey=a12b1234&t=" + title;
const http = context.services.get("IMDB"); // change the name of the service here if you used a different name.
return http
.get({ url: imdb_url })
.then(resp => {
var doc = EJSON.parse(resp.body.text());
movies.updateOne({"_id":docId}, doc);
});
};
As you can see in the middle of the code, I am using the OMDB API and I replaced the API key with a fake one.
You can create your own free API Key here with just a valid email address:
http://www.omdbapi.com/apikey.aspx
- limited to 1000 calls per day.
Once this is done, you can just replace my fake API key with your own key.
Now that this is ready, we just have to test it by repeating the same CURL command we used before. Feel free to use another movie title if you have a better one in mind ;-).
I removed the previous test we made before adding the trigger and now this is what I have in my collection:
Let’s review for a second what we have done:
We inserted a new document using an HTTP POST service hosted by MongoDB Stitch containing only an “_id” (automatically populated by MongoDB) and a “Title”.
The insertion of this document is detected by the trigger that calls an external Web API using this “Title” as the parameter.
We are then parsing the body of the result we get from this API and updating the document we inserted a few milliseconds ago.
Conclusion
With just a few lines of code, you can enrich the data where it lives with MongoDB Stitch. This is a great way to leverage your micro services architecture and finally switch to an event-driven architecture.
Next Steps
Thanks for taking the time to read my post. I hope you found it useful and interesting.
If MongoDB Stitch is something you are considering in production, you can discover here
how the billing works
.
If you want to query your data sitting in MongoDB Atlas using MongoDB Stitch, I recommend
this article from Michael Lynn
.
October 15, 2018