Store Images and Videos with Node.js, Integrating Azure Blob Storage, and MongoDB
27 min • Published Feb 12, 2024
Rate this video
00:00:00Introduction to MongoDB and Azure Blob Storage Integration
- Overview of the tutorial's goal to combine MongoDB with Azure Blob Storage.
- Prerequisites for the tutorial, including Node.js, MongoDB setup, and Azure storage account.00:01:47Setting Up Azure Blob Storage
- Step-by-step guide on creating a storage account in Azure.
- Instructions on selecting the right type of storage account and configuring advanced settings.
- Generating a shared access signature (SAS) for client access to data.00:07:50Preparing the Node.js Application
- Initializing the Node.js project with necessary packages.
- Setting up environment variables for MongoDB and Azure connections.
- Importing dependencies and establishing connections to MongoDB and Azure Blob Storage.00:15:10Handling Image Uploads
- Creating the server and defining the port for API calls.
- Writing the main function to handle image uploads, including metadata extraction and image streaming.00:22:30Correcting Errors and Running the Application
- Fixing typos and errors in the code to ensure proper execution.
- Running the Node.js application and verifying the upload of an image and metadata storage.00:27:34Conclusion and Additional Resources
- Recap of the tutorial's achievements and the benefits of using MongoDB with Azure Blob Storage.
- Encouragement to like, subscribe, and visit MongoDB's YouTube channel and developer center for more information.The primary focus of the video is to teach viewers how to create a Node.js API that uses MongoDB to store metadata and references to media files stored in Azure Blob Storage, showcasing how to utilize both services for effective media file management in web applications.
🔑 Key Points
- The tutorial demonstrates building a Node.js API for storing media files in Azure Blob Storage and referencing them in MongoDB.
- It covers setting up Azure storage accounts, generating shared access signatures, and configuring MongoDB connections.
- The video provides step-by-step coding instructions, including handling errors and correcting typos in the code.
- It emphasizes the importance of using both MongoDB and Azure Blob Storage to manage large files and metadata efficiently.
🔗 Related Links
Full Video Transcript
when storing media files such as images or videos a common question is how best to do this with your mongod be database hi I'm Tim and today we're going to build a node.js API that will allow you to store your media files in Azure blob storage while using mongod Tob to store reference to that media along with any Associated metadata allowing you to Leverage The Best of Both Technologies for this tutorial there's a few things you need already so first is no JS downloaded on your machine next is going to be AA to be close to your setup and lastly it's going to be an Azure storage account with an active subscription once you have all that you're ready to get started so what we're going to do is we're just going to pop up to the top on our Azure portal and we're going to type in storage now make sure you choose storage accounts and not storage accounts classic storage accounts classic doesn't have all the features we need for this tutorial so we go into storage accounts we're going to hit create and here we're just going to select our subscription and Resource Group so these are fine for me and we're going to give our counter name so this name has to be unique so silly little blob 1 two 3 it can be letters and numbers and everything here looks fine I'm just going to change the redundancy to the lowc cost option we're going to go into advanced now from here we're fine to accept the default options the thing to pay attention to is for Network I'm going to choose enable Public Access from all networks now this keeps it entirely open it allows us to bypass setting up access rules and it's just easier for this tutorial for production you might want to look into deciding who can access your API everything else here we're fine excep in the defaults and we're just going to scroll to the bottom and when it all looks good we're going to hit create now this will take a moment to deploy I'm going to pop back to you when everything's ready now that everything's set up we're just going to go to Resource so what we need to do is we need to set up container so you can think of a container like like a directory in a file system used for organizing the blops you can have as many containers as you need in your storage account and each container can hold numerous blobs so to do this we're just going to go to the left we're going to hit containers and we're going to hit create now we just need to give our container a name we'll just call it Blobby and create so now that we have that created we just need to get our SAS so our shared access signature this provides detail control over the ways your client can access your data so we'll pop back here to overview from the left we're going to go to Shared access signature we're going to hit object and everything else we're absolutely fine taking the defaults and we're just going to hit generate so what you need to copy down is this one here the second one SAS token we'll save that to clipboard and we're going to need it later when we're trying to access our blob from our application now that we have our Azure account set up we have mongod to be set up we're going to start with our application so to initialize everything you're going to go to your terminal and you're going to type mpm install and it's going to be hyphen hyphen save and we're going to go at Azure SL storage hyen blob and then it's going to be mongod DB andv so if we run this command here and we'll let that install so add a add a Jer storage blob package is for interacting with the Jer blob storage mongod Tob for the mongod Tob operations and DMV is for loading our environment variables we have everything going there so you'll see on the left side we have our new package.json and we also have our EMV file so our EMV is we're going to load our connection strings so first things first we'll get our Mong Tob connection string so if you go over to your mongodb cluster you hit connect drivers and here you'll see your connection string so we'll just hit copy and the one thing we'll need to change is the password so if we go back to our application we'll copy this in and then we'll just change our password now the next thing we'll need is we'll need our SAS token that we generated earlier so pop them back to your Azure account you're going to go back everything is still here and we'll just get our connection string so that's our SAS token if we hit copy on this we'll paste this in and then our account name and our container name that we set up earlier so our account name if you're a remember is silly little blob two3 just going to double check that's correct and then our container name you'll remember is Blobby perfect so we have everything here and we're going to pop over to our app so let's import our dependencies so if we hit import from HTTP or apologies import HTTP from HTTP perfect next we're going to import our blob service client so import report and this is blob service client from Azure for/ storage perfect we're going to get our client so import client import client perfect for mongodb and then lastly we're just going to import uhv config so we can access our environment variables so that's just EnV slig perfect now next steps here is to set up our environment variables so if we go cons and we'll get our mongod Tob Ur and that is equals to process. EnV do mongod DP underscore U perfect now we need to get our account name cons account name that's equal process. EnV dot account underscore name perfect and then our SAS token so const SAS token equals process. Envy do sasore token and then lastly our container name so const container name equals process. EnV do container name perfect so that's our all our environment variables loaded in and next thing to set up is is establishing our connection with the Zur blob storage so to do this we're going to need our blob service client and our container client so our con blob service client will'll get by using this here so that is equal to new blob service client and then in here we're going to have to put in a connection so that is tick and it's https and then it's for slash our account name and then it's going to be blob. core. windows.net and then forward slash and that's going to be our SAS token perfect so that's that there and then next thing is going to be our container client so that's just const container client equals blob service client do get container class and then just our container name now that's all we'll need to connect to our blob service account next is just to connect to our mongod to be so to connect to our mongod to be account it's just const client and then it's equals new client and then our Monga URI and then it's says client. connect perfect so that's everything we need to connect to the both of those and we'll look at uploading our file next so first things first we need to establish somewhere to make our API calls so we're going to create a HTTP server so const server and we'll use HTTP doc creat server and then we'll just put in handle image upload so this just means every time our server is called it's going to call the function handle image upload now we're going to decide what port it goes on so I'm just going to go const and Port equals 3,000 I'm going to fix this quick typo here and next we're just going to set our server to listen so server. listen and then it's port and our error function here perfect and once this is running we're going to pop in console log and then we'll just say what port it's listing on so server listening and then our Port perfect so now our server is set up to listen and next we're going to create that handle image upload function so this function is designed to process HTTP post requests to the for/ API upload endpoint handling the uploading of an image and storing its Associated metadata so we're going to create this function this function will call upon a couple of helper functions to ACH this we'll break down how they work as well just after we have all this written so we'll start with an asynchronous function that's handle image upload so handle image upload will take in the parameters now at the top of this function we're going to set the header for the response so set header and that's going to be content type and we're just going to set that to P Json so cont content hyphen type and then it's going to be application Json application SL Json perfect yeah place these with singles and everything looks good there next we're just going to contain all of our logic inside an IFL statement so this IFL statement we'll just make sure it's going to the right end point and that it is in fact the post request so if we'll check that it is going to the right URL first so URL and this will be our API SL upload so SL API SL upload make sure it's the correct destination and then on top of this we're going to check that the method is in fact post perfect and the actual logic will go in here and then the L statement will just write that it went to an incorrect endpoint so we'll write that first and that's just going to be response Dot wrad and that's going to be 404 and then we'll just return a response so do end and that's going to be json. stringify and then we'll just write error not found that seems plenty informative perfect now back to our if statement so what this will do is it will extract the metadata from the headers it will upload the image to Aero blob storage as a stream and it will store the metadata in mongodb so all of this will be inside of a TR catch block and inside our try it's going to be const and we're going to extract that metadata so that's going to be our file name it's going to be our caption and our file type so that seems like plenty of information to store and we're just going to go and we're going to call upon a function that we'll call extract metadata and we'll create that later so extract metadata and we'll have to hand in the headers to that one so that's perfect now that's how we get our metadata next is to upload the image so we're going to go const and we'll create an image URL to store in our manga to be so we'll call that image URL and what this will do is it will call upon another helper function that we'll just call await upload image streamed so this will just uploaded as a stream of bit so because it's being stored in binaries it will take care of that like synchronously so we'll call that file name and then just the request perfect and then last thing we need to do is we need to actually store our metadata in mongodb so what this will be is this will be a wait and we'll call this one store metadata so another helper function we'll set up and this will take in the file name it will take in the caption the file type and the image URL that we've generated from storing it in that blob storage perfect now last thing to do is just to return a positive response so that's res dot right head and this will be a 2011 everything went well and we'll just give some Json to that as well give a message so res. end and we'll go json. stringify and we'll just give a message that says go message and in this we can just go image uploaded metad dat stored perfect just as a string I think that should be plenty now we'll also write a catch for the error so we'll just conso log that error if anything does come up and that will just be error think it's good there and then we'll give a error response as well so res dot right head now this will be a 500 and in this we'll just give an error message as well json. end json. stringify and we'll write error and this will be a server error perfect now we have everything set up in this function that we need all that's left to do is to actually write these helper functions so now that we have our main function written it's time to handle our helper functions so we'll start with the first one extract metadata so that's just going to be another asynchronous function and what we're going to do for this one is we're going to pass in the headers so this where will assign values to our metadata that we'll store in mongodb so there's going to be a couple steps to this but first thing we're going to do is we're going to get the content type and this will be passed in straight away so we'll just go equals headers and we'll describe this as constant type or content type now content type will describe if it's an image if it's a video and it will also give the file type now in order to extract that to store in our database we're going to go const file type equals and we'll go content type and we're going to split this at the forward slash so when we're actually passing in our post request you'll be able to see where this is actually written and where we assign this value but it's going to be the second value in the array that we get out of that so the value app position one now after our file type we have a const disposition so that's our content disposition now what we'll do in this is we'll assign a name to the file that we're passing in will give information as to how we expect the server to handle it so in our case it will be an attachment so that expects the server to download it but depending on the application you might ask it to save it but that's where the description for it will go here and we're just going to go equals headers and we're going to go content we're going to pop all this in there and we're going to go content hyphen disposition perfect now now for this example we're going to pass in a caption so depending on your application you might want different fields but the caption will just be there to describe the actual image or video being stored we're going to keep that in headers we're going to say x image caption and if no caption is provided we'll just store that as a string and we'll say no caption provided perfect now after this we just have two more so we're going to have our const matches now now in this what we're doing is we're looking for a file name equals some file name that's how we'll assign the name to what we're storing in our database this line uses regular expression to find a file name in the content disposition string so the Rex is for SL file name and then equals quote and we're going to go bracket we'll go square bracket upper and then another single quote a square bracket again we're going to go plus and then close your brackets after that it's forward slash it's I to make it case insensitive and then we're going to go exec and we're going to run this on content disposition and that is everything we need here so now that we've extracted the file name from the curl or the from the post request we pass in last thing we need to do is we need to actually get the file name from that so we're going to go const file name and we're going to go equals we'll make sure that's uppercase so we'll go equals matches and we're just going to check to see if that regular expression actually matched and if it did we'll take the value that it comes from that that will be stored in the second place of the array so that's do one and if not we'll just generate an automatic name for the file and we'll do that by getting the we'll just use the date right now so that will be image and then we'll say date. now and then we'll also need to assign the file type so that will be what we got earlier so it's just file type and curly braces again perfect and that's our extract metadata function done so next next after that what we need to do is we need to get the upload image stream so I'll write that here and we'll just go just below it so for our next function it's upload image streamed and you'll see what we're going to be doing here is we're going to return an image URL as well of course I almost forgot we need to actually return our values so that's just going to be return and in here we're going to return our file name we're going to return our caption and we're going to return our file type perfect so now that we have that done we can get started on our next function so we're going to create our function async function and we'll call this upload image stream and for this we're going to pass in our blob name and our data stream perfect now we'll scroll down to give a bit of room on the screen so in this what we need for this is we need our blob client so we're just going to go const blob client and this will be a new value and we're going to create this using the container client doget so doget and block blob client and then this is where we need our blob name so this create a client for our blob now that we have that what we can do next is we can upload a stream so we're just going to go await and then we're going to go blob client and with this we'll use the upload stream and we'll pass in our data stream now what we need to do just for our metata is we need to return our blob client URL so this is just what we'll store in our mongod debe database so if we go return blob client and URL now we can access what's stored in our blob at this location perfect so we have our extract metadata we have our upload stream our last thing we just need is we need our store metadata so this is a function that will store our metadata in our mongod be database so if we just go async function and store metadata perfect and for this what we need to do is we need to pass in the name of what we're storing the caption the file type and the image URL that we just got from our blob so name caption file type and image URL now again if you're following along with this two tutorial you may want to use the same metadata Fields as I'm using but for your own applications you can store whatever data you need so for this we're just going to go con collection and we'll use client do DB we're going to go we're going to use the tutorial DB so this is just one that I have in my database and the collection metadata so just the database I want to use and the collection I want to use now if this database and this collection doesn't already exist there is no issue mongod Tob will automatically create it once you try to insert into that collection or database and what we need to do is we just need to actually insert into that so that's await collection and we're going to go insert one and we're going to actually store the data so we're just going to go name caption file type and then lastly the image URL perfect so if you followed along you should have everything you need to actually run this application we're going to get to testing now okay now before we can actually run our application there are a few areas we need to correct in it so what we'll do is we'll start at the top of the application and you'll see here when I was def finding the container client I accidentally called it get container class that should be get container client below that we need for our client we need to have that be a method so just add the brackets at the end of that and underneath all of this we need to go and we need to make sure that for the extract metadata we're actually passing in the headers so that's rec. headers and after that we just need to make sure that all of our variables are named correctly so for file name it should be file and then name with capital N so that's so everything is camel case everything is named correctly the last last thing we need to do is we need to make sure our extract metadata is asynchronous so that is just await and we can save everything there and run our application so we're just going to pop down to the terminal and we're going to type node then app. MJS and perfect so our server is listing on Port 3000 so if we go back to our terminal and we're just going to run our Command that we had from earlier and you'll see perfect okay so message and it's image uploaded metadata stored we're just going to have a look at that in our blob and in our mongod to be collection so first of all we'll refresh this we have Mong to be as webs scale.png and if we go into the best way to look at that is usually just edit we can see our image is a lovely screenshot of a teddy with mang to be is web scale and if we want to go to our metadata again we'll refresh our collection as well and we look at this perfect so we have the name we have the caption we have the file type and the URL through this tutorial we demonstrated how to build a node.js API to seems to interact with MBE and aure blob storage to effectively store your media files utilizing the powerful capabilities of both Technologies developers can build applications such as social media apps or content Management Systems if you like this video or find it useful make sure to like And subscribe so you don't miss our upcoming tutorials and if you want to learn more about manga to be visit our YouTu YouTube channel or a developer Center