Maxime Beugnet

14 results

Data Enrichment with MongoDB Stitch

Objectives Here is what we are going to achieve in this tutorial: Firstly, we are going to write a document to MongoDB using MongoDB Stitch . The result in our MongoDB collection will look like this: { "_id": ObjectId("5bb27712dced5f37bebf388c"), "Title":"Guardians of the Galaxy" } Secondly, a trigger will catch this new insertion and start a function. Lastly, this function will call the OMDB external API with the given movie title, fetch data about that movie, and finally enrich our MongoDB document with the data we gathered from this API. This is the final result we expect in our MongoDB collection: { "_id": ObjectId("5bb27712dced5f37bebf388c"), "Title":"Guardians of the Galaxy", "Year":"2014", "Rated":"PG-13", "Released":"01 Aug 2014", "Runtime":"121 min", "Genre":"Action, Adventure, Comedy", "Director":"James Gunn", "Writer":"James Gunn, Nicole Perlman, Dan Abnett (based on the Marvel comics by), Andy Lanning (based on the Marvel comics by), Bill Mantlo (character created by: Rocket Raccoon), Keith Giffen (character created by: Rocket Raccoon), Jim Starlin (characters created by: Drax the Destroyer, Gamora & Thanos), Steve Englehart (character created by: Star-Lord), Steve Gan (character created by: Star-Lord), Steve Gerber (character created by: Howard the Duck), Val Mayerik (character created by: Howard the Duck)", "Actors":"Chris Pratt, Zoe Saldana, Dave Bautista, Vin Diesel", "Plot":"A group of intergalactic criminals are forced to work together to stop a fanatical warrior from taking control of the universe.", "Language":"English", "Country":"USA", "Awards":"Nominated for 2 Oscars. Another 52 wins & 99 nominations.", "Poster":"https://m.media-amazon.com/images/M/MV5BMTAwMjU5OTgxNjZeQTJeQWpwZ15BbWU4MDUxNDYxODEx._V1_SX300.jpg", "Ratings":[ { "Source":"Internet Movie Database", "Value":"8.1/10" }, { "Source":"Rotten Tomatoes", "Value":"91%" }, { "Source":"Metacritic", "Value":"76/100" } ], "Metascore":"76", "imdbRating":"8.1", "imdbVotes":"871,949", "imdbID":"tt2015381", "Type":"movie", "DVD":"09 Dec 2014", "BoxOffice":"$270,592,504", "Production":"Walt Disney Pictures", "Website":"http://marvel.com/guardians", "Response":"True" } Prerequisites So first of all, if you want to try this at home, it is very easy. The only requirement here is to create a free MongoDB Atlas cluster. This video will show you the steps. MongoDB Stitch is our serverless platform, built by MongoDB on top of MongoDB Atlas. Once our MongoDB Atlas cluster is ready to use, link a MongoDB Stitch application to it: Click on the left panel on “Stitch Apps”, Then click on “Create New Application”, Pick the name you want for your application, Link it to your free MongoDB Atlas cluster. Actions To be able to send a document to MongoDB, we are going to use an HTTP POST service. On the left panel, click on “Services”, Then click on “Add a Service”, Choose a service name “IMDB”, Note: “IMDB” will be reuse later in the function code. If you choose another name, please make sure to update the code accordingly. Click on “Add Service”, Click on “Add Incoming Webhook”, And copy the screenshot below. When this is done, hit the “Save” button and you will be in the “Function Editor” screen. Enter the following code: exports = function(payload, response) { const mongodb = context.services.get("mongodb-atlas"); const movies = mongodb.db("stitch").collection("movies"); var body = EJSON.parse(payload.body.text()); movies.insertOne(body) .then(result => { response.setStatusCode(201); }); }; Click the “Save” button again. Now that our Service is ready, we can test it! Go to the “Settings” and you will find your Webhook URL. You can now send an HTTP POST request like this to MongoDB Stitch: curl -H "Content-Type: application/json" -d '{"Title":"Guardians of the Galaxy"}' https://webhooks.mongodb-stitch.com/api/client/v2.0/app/stitchtapp-abcde/service/IMDB/incoming_webhook/post_movie_title?secret=test Note: I used a curl command but feel free to use Postman or whatever you are used to. We can check it worked by having a look at the content of the “stitch.movies” collection in our MongoDB Atlas Cluster: Now that we can insert a new document into MongoDB Atlas using Stitch, we are going to create the trigger. On the left panel click on “Triggers”, Then click on “Add a Database Trigger”, And do as follow. This is the code you will need to create the new function: exports = function(changeEvent) { var docId = changeEvent.documentKey._id; var title = encodeURIComponent(changeEvent.fullDocument.Title.trim()); var movies = context.services.get("mongodb-atlas").db("stitch").collection("movies"); var imdb_url = "http://www.omdbapi.com/?apikey=a12b1234&t=" + title; const http = context.services.get("IMDB"); // change the name of the service here if you used a different name. return http .get({ url: imdb_url }) .then(resp => { var doc = EJSON.parse(resp.body.text()); movies.updateOne({"_id":docId}, doc); }); }; As you can see in the middle of the code, I am using the OMDB API and I replaced the API key with a fake one. You can create your own free API Key here with just a valid email address: http://www.omdbapi.com/apikey.aspx - limited to 1000 calls per day. Once this is done, you can just replace my fake API key with your own key. Now that this is ready, we just have to test it by repeating the same CURL command we used before. Feel free to use another movie title if you have a better one in mind ;-). I removed the previous test we made before adding the trigger and now this is what I have in my collection: Let’s review for a second what we have done: We inserted a new document using an HTTP POST service hosted by MongoDB Stitch containing only an “_id” (automatically populated by MongoDB) and a “Title”. The insertion of this document is detected by the trigger that calls an external Web API using this “Title” as the parameter. We are then parsing the body of the result we get from this API and updating the document we inserted a few milliseconds ago. Conclusion With just a few lines of code, you can enrich the data where it lives with MongoDB Stitch. This is a great way to leverage your micro services architecture and finally switch to an event-driven architecture. Next Steps Thanks for taking the time to read my post. I hope you found it useful and interesting. If MongoDB Stitch is something you are considering in production, you can discover here how the billing works . If you want to query your data sitting in MongoDB Atlas using MongoDB Stitch, I recommend this article from Michael Lynn .

October 15, 2018

Capture IOT Data With MongoDB Stitch in 5 Minutes

Capturing IOT data is a complex task for 2 main reasons: We have to deal with a huge amount of data so we need a rock solid architecture. While keeping a bulletproof security level. First, let’s have a look at a standard IOT capture architecture: On the left, we have our sensors. Let’s assume they can push data every second over TCP using a POST and let’s suppose we have a million of them. We need an architecture capable to handle a million queries per seconds and able to resist any kind of network or hardware failure. TCP queries need to be distributed evenly to the application servers using load balancers and finally, the application servers are able to push the data to our multiple Mongos routers from our MongoDB Sharded Cluster . As you can see, this architecture is relatively complex to install. We need to: buy and maintain a lot of servers, make security updates on a regular basis of the Operating Systems and applications, have an auto-scaling capability (reduce maintenance cost & enable automatic failover)... This kind of architecture is expensive and maintenance cost can be quite high as well. Now let’s solve this same problem with MongoDB Stitch! Once you have created a MongoDB Atlas cluster , you can attach a MongoDB Stitch application to it and then create an HTTP Service containing the following code: exports = function(payload, response) { const mongodb = context.services.get("mongodb-atlas"); const sensors = mongodb.db("stitch").collection("sensors"); var body = EJSON.parse(payload.body.text()); body.createdAt = new Date(); sensors.insertOne(body) .then(result => { response.setStatusCode(201); }); }; And that’s it! That’s all we need! Our HTTP POST service can be reached directly by the sensors from the webhook provided by MongoDB Stitch like so: curl -H "Content-Type: application/json" -d '{"temp":22.4}' https://webhooks.mongodb-stitch.com/api/client/v2.0/app/stitchtapp-abcde/service/sensors/incoming_webhook/post_sensor?secret=test Because MongoDB Stitch is capable of scaling automatically according to demand, you no longer have to take care of infrastructure or handling failovers. Next Step Thanks for taking the time to read my post. I hope you found it useful and interesting. If you are looking for a very simple way to get started with MongoDB, you can do that in just 5 clicks on our MongoDB Atlas database service in the cloud. You can also try MongoDB Stitch for free and discover how the billing works . If you want to query your data sitting in MongoDB Atlas using MongoDB Stitch, I recommend this article from Michael Lynn .

October 3, 2018