MongoDB

Tim Kelly

Developer Advocate

You're reaching your limits. Your application is taking on massive amounts of data each day. More features, more interactions, everything you wanted your application to achieve, but you've reached a bottleneck. With an application with high throughput, sharding might be the solution you're looking for.

[Sharding][1] is a method for distributing a dataset across multiple machines. Databases with large enough datasets or read/write throughput can challenge the capacity of any one server. There are two methods to address this challenge: *vertical scaling and horizontal scaling.*

*Vertical scaling* is straightforward. Query rates are exhausting the CPU? Upgrade the CPU! My working sets have exceeded the size of the system's RAM? More RAM! While upgrading your server is a perfectly valid and oftentimes correct solution, there are limits. Hardware only gets so good, thanks to limits on the technology. And oftentimes, the increasing cost is exponential compared to the return on your investment, so you hit a point of diminishing returns.

*Horizontal scaling* means dividing your dataset and workload over multiple servers (such as through sharding). Each machine may only be as powerful as the last but only handles a subset of the overall workload, potentially providing better efficiency than one super-powered server. *"Many hands make light work!"* 

The trade-off is an increased complexity in infrastructure but expanding capacity means only adding new servers as needed, and MongoDB Atlas reduces a lot of the complexity in managing sharded clusters.

## Prerequisites
- A [MongoDB account][2] 
    - You will need a cluster [M30+ to enable sharding][3] (we’ll set this up later).
- [Java 17+][4]
- [Maven (version 3.9.6+)][5]
- A Spring Boot Project with
	- Spring Data MongoDB
	- Spring Web
	- You can set this up using [Spring Initializr](https://start.spring.io/)
- [MongoDB Shell](https://www.mongodb.com/try/download/shell)

## Set up MongoDB Atlas sharding
Create a sharded cluster, M30 or above. When creating your cluster, in additional settings, select **Sharding** and toggle to on. Set the number of shards to deploy with the sharded cluster. For production applications, use more than one shard or you will not be able to reap the full benefits of your sharded cluster, which you can read more about in the [sharding documentation][6]. This can be between 1 and 100 shards, inclusive. Wait for your cluster to be created. This will take a few minutes, so get a cup of coffee or scroll your TikToks. Once it is set up, select **load the sample data**, for this tutorial.

Spring Data MongoDB does not automatically set up sharding for collections. You need to perform these operations manually. We can do this using `mongosh`.

Connect to your MongoDB cluster as an admin with your connection string via `mongosh`:
```bash
mongosh "mongodb+srv://<username>:<password>@<cluster-url>/admin"
```

Shard the users collection by email:
```bash
sh.shardCollection("sample_mflix.users", { email: 1 })
```

You can verify that the sharding is enabled and the collection is sharded by running `sh.status()`.

### Define your entity with sharding in Spring Data
In your Spring Boot project, define an entity class and use the `@Sharded` annotation to specify the shard key fields.  

Spring Data MongoDB uses the `@Sharded` annotation to identify entities stored in sharded collections.

```java
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;
import org.springframework.data.mongodb.core.mapping.Field;
import org.springframework.data.mongodb.core.mapping.Sharded;

@Document("users")
@Sharded(shardKey = { "email" }) 
public class User {

    @Id
    private String id;
    private String name;
    @Field("email")
    private String email;
    private String password;

    // Getters and Setters
}
```
Adding the `@Sharded` annotation to an entity helps Spring Data MongoDB optimize operations in a sharded environment. It ensures that the shard key is included in `replaceOne` queries during upserts. Sometimes, this might need an extra check to find the current shard key value.

### Create a MongoDB repository
Define a repository interface for your `User` entity by extending `MongoRepository`.
```java
import org.springframework.data.mongodb.repository.MongoRepository;

import com.mongodb.sharded.model.User;

public interface UserRepository extends MongoRepository<User, String> {
}
```

### Create a service layer
Create a service class to handle business logic and interact with the repository.
```java
import java.util.List;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;

import com.mongodb.sharded.model.User;
import com.mongodb.sharded.repository.UserRepository;

@Service
public class UserService {

    @Autowired
    private UserRepository userRepository;

    public List<User> getAllUsers() {
        return userRepository.findAll();
    }

    public User saveUser(User user) {
        return userRepository.save(user);
    }
}
```

### Create a controller layer
Create a controller class to expose REST endpoints for the `User` entity.
```java
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.*;

import java.util.List;

@RestController
@RequestMapping("/users")
public class UserController {

    @Autowired
    private UserService userService;

    @GetMapping
    public List<User> getAllUsers() {
        return userService.getAllUsers();
    }

    @PostMapping
    public User createUser(@RequestBody User user) {
        return userService.saveUser(user);
    }
}
```
### Configure Spring Boot application
Ensure your Spring Boot application is configured to connect to MongoDB Atlas. In your application.properties or `application.yml`, provide the MongoDB connection URI.
```xml
spring.data.mongodb.uri=mongodb+srv://<username>:<password>@<cluster-url>/myDatabase?retryWrites=true&w=majority
```
Replace `<username>`, `<password>`, and `<cluster-url>` with your MongoDB Atlas credentials and connection string.

### Best practices
Choose a sharding key that ensures an even distribution of data. The [choice of a sharding key][7] is critical to the performance and scalability of your sharded cluster. An ideal sharding key should:

 - **Have high cardinality:** This means the key should have many unique
   values to ensure data is evenly distributed across shards. It is not
   necessary that a sharding key be entirely unique. 
 - **Provide even distribution:** The key should distribute documents evenly across all
   shards to avoid hotspots where one shard handles more data or
   requests than others. 
 - **Support common queries:** Choose a key that
   aligns with your most common query patterns to minimize query scatter
   and optimize performance.

For the users collection in the `sample_mflix` database, using the email field as the sharding key can be a good choice if:
 - Emails are unique and well-distributed.
 - Queries frequently filter or sort by email.

## Conclusion
Sharding with MongoDB Atlas and Spring Boot allows your application to scale efficiently and handle high loads with improved performance and availability. By following this guide, you can set up a sharded database, develop microservices, and ensure your application is ready to grow and succeed in a demanding environment.  

If you want to learn more about MongoDB and Java, read more on Developer Center, where you can learn how to deploy your sharded spring application as a [microservice with Azure and Kubernetes][8]. Or if you have any questions, head over to our [MongoDB Community forums][9].


  [1]: https://www.mongodb.com/docs/manual/sharding/
  [2]: https://www.mongodb.com/cloud/atlas/register
  [3]: https://www.mongodb.com/docs/atlas/cluster-additional-settings/#std-label-create-cluster-sharding
  [4]: https://www.oracle.com/java/technologies/downloads/
  [5]: https://maven.apache.org/download.cgi
  [6]: https://www.mongodb.com/docs/atlas/cluster-additional-settings/#std-label-create-cluster-sharding
  [7]: https://www.mongodb.com/docs/manual/core/sharding-choose-a-shard-key/
  [8]: https://www.mongodb.com/developer/languages/java/azure-kubernetes-services-java-microservices/
  [9]: https://www.mongodb.com/community/

Learn how to incorporate sharding into your Spring Boot App with Spring Data MongoDB. Improve the efficiency of your high throughput applications with horizontal scaling. 

Bring Sharding to Your Spring Boot App with Spring Data MongoDB

Unlocking Semantic Search: Building a Java-Powered Movie Search Engine with Atlas Vector Search and Spring Boot

## Introduction
In the early days of software development, application development consisted of monolithic codebases. With challenges in scaling, singular points of failure, and inefficiencies in updating, a solution was proposed. A modular approach. A symphony of applications managing their respective domains in harmony. This is achieved using microservices.

Microservices are an architectural approach that promotes the division of applications into smaller, loosely coupled services. This allows application code to be delivered in manageable pieces, independent of each other. These services operate independently, addressing a lot of the concerns of monolithic applications mentioned above.

While each application has its own needs, microservices have proven themselves as a viable solution time and time again, as you can see in the success of the likes of Netflix.

In this tutorial, we are going to deploy a simple Java Spring Boot microservice application, hosted on the [Azure Kubernetes Service (AKS)](https://learn.microsoft.com/en-us/azure/aks/intro-kubernetes). AKS simplifies deploying a managed Kubernetes cluster in Azure by offloading the operational overhead to Azure. We'll explore containerizing our application and setting up communication between our APIs, a MongoDB database, and the external world. You can access the full code here:

```bash
git clone https://github.com/mongodb-developer/simple-movie-microservice.git
```

Though we won't dive into the most advanced microservice best practices and design patterns, this application gives a simplistic approach that will allow you to write reviews for the movies in the MongoDB sample data, by first communicating with the review API and that service, verifying that the user and the movie both exist. The architecture will look like this.

![The architecture diagram for the microservice application.][1]

Simple, right? Exactly! And AKS makes it even easier to communicate with other APIs managed on the service. In a microservices architecture, services need to discover each other dynamically. Kubernetes itself acts as a service registry, where each service in the cluster has a unique name. So when we communicate with our other services in [ReviewController.java](https://github.com/mongodb-developer/simple-movie-microservice/blob/main/reviews/src/main/java/com/example/reviews/controller/ReviewController.java), we simply send a request to `http://user-management-service/users/`. In this demo application, communication is done with RESTful HTTP/S requests, using [RestTemplate](https://www.baeldung.com/rest-template). 

## Prerequisites
Before you begin, you'll need a few prerequisites to follow along with this tutorial, including:
- A [MongoDB Atlas](https://www.mongodb.com/cloud/atlas) account, if you don't have one already, with a cluster ready with the [MongoDB sample data](https://www.mongodb.com/basics/sample-database).
- A [Microsoft Azure account](https://azure.microsoft.com/en-us/free) with an active subscription.
- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/), or you can [install Azure PowerShell](https://learn.microsoft.com/en-us/powershell/azure/), but this tutorial uses Azure CLI. Sign in and configure your command line tool following the steps in the documentation for [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/azure-cli-configuration) and [Azure PowerShell](https://learn.microsoft.com/en-us/powershell/azure/configure-global-settings).
- [Docker](https://www.docker.com/get-started/) for creating container images of our microservices.
- Java 17.
- Maven 3.9.6.

## Set up an Azure Kubernetes Service cluster
Starting from the very beginning, set up an Azure Kubernetes Service (AKS) cluster.

### Install kubectl and create an AKS cluster
Install `kubectl`, the Kubernetes command-line tool, via the Azure CLI with the following command (you might need to sudo this command), or you can download the binaries from:
```bash
az aks install-cli
```

Log into your Azure account using the Azure CLI:
```bash
az login
```

Create an Azure Resource Group:
```bash
az group create --name myResourceGroup --location northeurope
```

Create an AKS cluster: Replace `myAKSCluster` with your desired cluster name. (This can take a couple of minutes.)
```bash
az aks create --resource-group myResourceGroup --name myAKSCluster --node-count 2 --enable-addons monitoring --generate-ssh-keys
```

### Configure kubectl to use your AKS cluster
After successfully creating your AKS cluster, you can proceed to configure `kubectl` to use your new AKS cluster. Retrieve the credentials for your AKS cluster and configure `kubectl`:
```bash
az aks get-credentials --resource-group myResourceGroup --name myAKSCluster
```

### Create an Azure Container Registry (ACR)
Create an ACR to store and manage container images across all types of Azure deployments:
```bash
az acr create --resource-group <your-resource-group> --name <your-container-registry> --sku Basic
```
> Note: Save the app service id here. We’ll need it later when we are creating a service principal.

Log into ACR:
```bash
az acr login --name <your-container-registry>
```
## Containerize your microservices application
Each of your applications (User Management, Movie Catalogue, Reviews) has a `Dockerfile`. Create a .jar by running the command `mvn package` for each application, in the location of the pom.xml file. Depending on your platform, the following steps are slightly different.

For those wielding an M1 Mac, a bit of tweaking is in order due to our image's architecture. As it stands, Azure Container Apps can only jive with linux/amd64 container images. However, the M1 Mac creates images as `arm` by default. To navigate this hiccup, we'll be leveraging Buildx, a handy Docker plugin. Buildx allows us to build and push images tailored for a variety of platforms and architectures, ensuring our images align with Azure's requirements.

### Build the Docker image (not M1 Mac)
To build your image, make sure you run the following command in the same location as the `Dockerfile`. Repeat for each application.
```bash
docker build -t movie-catalogue-service .
```
**Or** you can run the following command from the simple-movie-microservice folder to loop through all three repositories.

```bash
for i in movie-catalogue reviews user-management; do cd $i; ./mvnw clean package; docker build -t $i-service .; cd -; done
```
### Build the Docker image (M1 Mac)
If you are using an M1 Mac, use the following commands to use Buildx to create your images:
```bash
docker buildx install
```

Next, enable Buildx to use the Docker CLI:
```bash
docker buildx create --use
```

Open a terminal and navigate to the root directory of the microservice where the `Dockerfile` is located. Run the following command to build the Docker image, replacing `movie-catalogue-service` with the appropriate name for each service.
```bash
docker buildx build --platform linux/amd64 -t movie-catalogue-service:latest --output type=docker .
```

### Tag and push
Now, we're ready to tag and push your images. Replace `<your-acr-name>` with your actual ACR name. Repeat these two commands for each microservice. 
```bash
docker tag movie-catalogue-service <your-acr-name>.azurecr.io/movie-catalogue-service:latest
docker push <your-acr-name>.azurecr.io/movie-catalogue-service:latest
```
**Or** run this script in the terminal, like before:

```bash
ACR_NAME="<your-acr-name>.azurecr.io"

for i in movie-catalogue reviews user-management; do 
    # Tag the Docker image for Azure Container Registry
    docker tag $i-service $ACR_NAME/$i-service:latest
    # Push the Docker image to Azure Container Registry
    docker push $ACR_NAME/$i-service:latest
done
```

## Deploy your microservices to AKS
Now that we have our images ready, we need to create Kubernetes deployment and service YAML files for each microservice. We are going to create one *mono-file* to create the Kubernetes objects for our deployment and services. We also need one to store our MongoDB details. It is good practice to use secrets for sensitive data like the MongoDB URI.

### Create a Kubernetes secret for MongoDB URI
First, you'll need to create a secret to securely pass the MongoDB connection string to your microservices. In Kubernetes, the data within a secret object is stored as [base64-encoded](https://en.wikipedia.org/wiki/Base64) strings. This encoding is used because it allows you to store binary data in a format that can be safely represented and transmitted as plain text. It's not a form of encryption or meant to secure the data, but it ensures compatibility with systems that may not handle raw binary data well.

Create a Kubernetes secret that contains the MongoDB URI and database name. You will encode these values in Base64 format, but Kubernetes will handle them as plain text when injecting them into your [pods](https://kubernetes.io/docs/concepts/workloads/pods/). You can encode them with the bash command, and copy them into the YAML file, next to the appropriate data keys:

```bash
echo -n 'your-mongodb-uri' | base64
echo -n 'your-database-name' | base64
```

This is the [mongodb-secret.yaml](https://github.com/mongodb-developer/simple-movie-microservice/blob/main/mongodb-secret.yaml).
```yaml
apiVersion: v1
kind: Secret
metadata:
  name: mongodb-secret
type: Opaque
data:
  MONGODB_URI: <Base64 encoded MongoDB URI>
  MONGODB_DATABASE: <Base64 encoded MongoDB database name>

```

Run the following command to apply your secrets:
```bash
kubectl apply -f mongodb-secret.yaml
```

So, while base64 encoding doesn't secure the data, it formats it in a way that's safe to store in the Kubernetes API and easy to consume from your applications running in pods.

### Authorize access to the ACR
If your ACR is private, you'll need to ensure that your Kubernetes cluster has the necessary credentials to access it. You can achieve this by creating a Kubernetes secret with your registry credentials and then using that secret in your deployments. 

The next step is to create a service principal or use an existing one that has access to your ACR. This service principal needs the `AcrPull` role assigned to be able to pull images from the ACR. Replace  `<your-service-principal-name>`, `<your-subscription-id>`, `<your-resource-group-name>`, and `<your-acr-name>` with your own values.
<your-service-principal-name>: This can be any unique identifier you want to give this service principal.
<your-subscription-id>: You can get the id for the subscription you’re using with `az account show --query id --output tsv`.
<your-resource-group-name>: Use the same resource group you have your AKS set up in.
<your-acr-name>: This is the Azure Container Registry you have your images stored in.

```bash
az ad sp create-for-rbac --name <your-service-principal-name> --role acrPull --scopes /subscriptions/<your-subscription-id>/resourceGroups/<your-resource-group-name>/providers/Microsoft.ContainerRegistry/registries/<your-acr-name>
```

This command will output JSON that looks something like this:
```bash
{   
	"appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",   
	"displayName": "<your-service-principal-name>",   
	"password": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",   
	"tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" 
}
```

- `appId` is your `<your-service-principal-app-id>`.
- `password` is your `<your-service-principal-password>`.

**Note:** It's important to note that the `password` is only displayed once at the creation time. Make sure to copy and secure it.

**Create a Kubernetes secret with the service principal's credentials.** You can do this with the following command:

```bash
kubectl create secret docker-registry acr-auth \
  --namespace default \
  --docker-server=<your-acr-name>.azurecr.io \
  --docker-username=<your-service-principal-app-id> \
  --docker-password=<your-service-principal-password> \
  --docker-email=<your-email>
```

### Create Kubernetes deployment and service YAML files
There are a couple of points to note in the [YAML file](https://github.com/mongodb-developer/simple-movie-microservice/blob/main/all-microservices.yaml) for this tutorial, but these points are not exhaustive of everything happening in this file. If you want to learn more about configuring your YAML for Kubernetes, check out the documentation for [configuring Kubernetes objects](https://kubernetes.io/docs/concepts/overview/working-with-objects/).
- We will have our APIs exposed externally. This means you will be able to access the endpoints from the addresses we'll receive when we have everything running. Setting the `type: LoadBalancer` triggers the cloud provider's load balancer to be provisioned automatically. The external load balancer will be configured to route traffic to the Kubernetes service, which in turn routes traffic to the appropriate pods based on the service's selector.
- The `containers:` section defines a single container named `movie-catalogue-service`, using an image specified by `<your-container-registry>/movie-catalogue-service:latest`.
- `containerPort: 8080` exposes port 8080 inside the container for network communication.
- Environment variables `MONGODB_URI` and `MONGODB_DATABASE` are set using values from secrets (`mongodb-secret`), enhancing security by not hardcoding sensitive information.
- `imagePullSecrets: - name: acr-auth` allows Kubernetes to authenticate to a private container registry to pull the specified image, using the secret we just created.

```yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: movie-catalogue-service-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: movie-catalogue-service
  template:
    metadata:
      labels:
        app: movie-catalogue-service
    spec:
      containers:
      - name: movie-catalogue-service
        image: <your-container-registry>/movie-catalogue-service:latest
        ports:
        - containerPort: 8080
        env:
        - name: MONGODB_URI
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: MONGODB_URI
        - name: MONGODB_DATABASE
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: MONGODB_DATABASE
      imagePullSecrets:
      - name: acr-auth
---
apiVersion: v1
kind: Service
metadata:
  name: movie-catalogue-service
spec:
  selector:
    app: movie-catalogue-service
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer
---
```

Remember, before applying your Kubernetes YAML files, make sure your Kubernetes cluster has access to your ACR. You can configure this by granting AKS the ACRPull role on your ACR:

```bash
az aks update -n <your-AKS-Cluster> -g <your-resource-group> --attach-acr <your-ACR>
```

Replace `<your-AKS-Cluster>`, `<your-resource-group>`, and `<your-ACR>` with your AKS cluster name, Azure resource group name, and ACR name, respectively.

### Apply the YAML file
Apply the YAML file with `kubectl`:
```bash
kubectl apply -f all-microservices.yaml
```
## Access your services
Once deployed, it may take a few minutes for the LoadBalancer to be provisioned and for the external IP addresses to be assigned. You can check the status of your services with:
```bash 
kubectl get services
```

Look for the external IP addresses for your services and use them to access your microservices.

After deploying, ensure your services are running:
```bash
kubectl get pods
```
Access your services based on the type of Kubernetes service you've defined (e.g., LoadBalancer in our case) and perform your tests.

You can test if the endpoint is running with the CURL command:
```bash
curl -X POST http://<ReviewAPI-External-IP>/reviews \
     -H "Content-Type: application/json" \
     -d '{"movieId": "573a1391f29313caabcd68d0", "userId": "59b99db5cfa9a34dcd7885b8", "rating": 4}'
```

And this review should now appear in your database. You can check with a simple:
```bash
curl -X GET http://<ReviewAPI-External-IP>/reviews
```

Hooray!

## Conclusion
As we wrap up this tutorial, it's clear that embracing microservices architecture, especially when paired with the power of Kubernetes and Azure Kubernetes Service (AKS), can significantly enhance the scalability, maintainability, and deployment flexibility of applications. Through the practical deployment of a simple microservice application using Java Spring Boot on AKS, we've demonstrated the steps and considerations involved in bringing a microservice architecture to life in the cloud.

Key takeaways:
- **Modular approach**: The transition from monolithic to microservices architecture facilitates a modular approach to application development, enabling independent development, deployment, and scaling of services.
- **Simplified Kubernetes deployment**: AKS abstracts away much of the complexity involved in managing a Kubernetes cluster, offering a streamlined path to deploying microservices at scale.
- **Inter-service communication**: Utilizing Kubernetes' internal DNS for service discovery simplifies the communication between services within a cluster, making microservice interactions more efficient and reliable.
- **Security and configuration best practices**: The tutorial underscored the importance of using Kubernetes secrets for sensitive configurations and the Azure Container Registry for securely managing and deploying container images.
- **Exposing services externally**: By setting services to `type: LoadBalancer`, we've seen how to expose microservices externally, allowing for easy access and integration with other applications and services.

The simplicity and robustness of Kubernetes, combined with the scalability of AKS and the modularity of microservices, equip developers with the tools necessary to build complex applications that are both resilient and adaptable. If you found this tutorial useful, find out more about what you can do with MongoDB and Azure on our [Developer Center](https://www.mongodb.com/developer/technologies/azure/).

Are you ready to start building with Atlas on Azure? Get started for free today with [MongoDB Atlas on Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/mongodb.mongodb_atlas_self_serve_prod_2022).


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7bcbee53fc14653b/661808089070f0c0a8d50771/AKS_microservices.png

Learn how to deploy your Java Spring Boot microservice to Azure Kubernetes Services.

Using Azure Kubernetes Services for Java Spring Boot Microservices

Using Microsoft Entra ID, Spring Boot Security, and Spring Data MongoDB, make a secure rest API.

Secure your API with Spring Data MongoDB and Microsoft EntraID

For as long as you have been reviewing restaurants, you've been storing your data in MongoDB. The plethora of data you've gathered is so substantial, you decide to team up with your friends to host this data online, so other restaurant goers can decide where to eat, informed by your detailed insights. But your friend has been storing their data in Azure Blob storage. They use JSON now, but they have reviews upon reviews stored as `.csv` files. How can we get all this data pooled together without the often arduous process of migrating databases or transforming data? With MongoDB's Data Federation, you can combine all your data into one unified view, allowing you to easily search for the best French diner in your borough. 

This tutorial will walk you through the steps of combining your MongoDB database with your Azure Blob storage, utilizing MongoDB's Data Federation.

## Prerequisites
Before you begin, you'll need a few prerequisites to follow along with this tutorial, including:
- A [MongoDB Atlas](https://www.mongodb.com/cloud/atlas?tck=docs_atlas) account, if you don't have one already
- A [Microsoft Azure account](https://azure.microsoft.com/en-us/free) with a storage account and container setup. If you don't have this, follow the steps in the Microsoft documentation for the [storage account](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal) and the [container](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal).
- [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/), or you can [install Azure PowerShell](https://learn.microsoft.com/en-us/powershell/azure/), but this tutorial uses Azure CLI. Sign in and configure your command line tool following the steps in the documentation for [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/azure-cli-configuration) and [Azure PowerShell](https://learn.microsoft.com/en-us/powershell/azure/configure-global-settings).
- Node.js 18 or higher and npm: Make sure you have [Node.js and npm (Node.js package manager)](https://nodejs.org/en/download) installed. Node.js is the runtime environment required to run your JavaScript code server-side. npm is used to manage the dependencies.

### Add your sample data
To have something to view when your data stores are connected, let's add some reviews to your blob. First, you'll add a review for a new restaurant you just reviewed in Manhattan. Create a file called example1.json, and copy in the following:

```json
{
	"address":{
		"building":"518",
		"coord":[
			{
				"$numberDouble":"-74.006220"
			},
			{
				"$numberDouble":"40.733740"
			}
		],
		"street":"Hudson Street",
		"zipcode":"10014"
	},
	"borough":"Manhattan",
	"cuisine": [
		"French",
		"Filipino"
	],
	"grades":[
		{
			"date":{
				"$date":{
					"$numberLong":"1705403605904"
				}
			},
			"grade":"A",
			"score":{
				"$numberInt":"12"
			}
		}
	],
	"name":"Justine's on Hudson",
	"restaurant_id":"40356020"
}
```

Upload this file as a blob to your container:
```bash
az storage blob upload --account-name <YourStorageAccountName> --container-name <YourContainerName> --name <BlobName> --file <PathToFile>
```

Here, `BlobName` is the name you want to assign to your blob (just use the same name as the file), and `PathToFile` is the path to the file you want to upload (example1.json).

But you're not just restricted to JSON in your federated database. You're going to create another file, called example2.csv. Copy the following data into the file:

```csv
Restaurant ID,Name,Cuisine,Address,Borough,Latitude,Longitude,Grade Date,Grade,Score
40356030,Sardi's,Continental,"234 W 44th St, 10036",Manhattan,40.757800,-73.987500,1927-09-09,A,11

```

Load example2.csv to your blob using the same command as above.

You can list the blobs in your container to verify that your file was uploaded:

```bash
az storage blob list --account-name <YourStorageAccountName> --container-name <YourContainerName> --output table
```

## Connect your databases using Data Federation
The first steps will be getting your MongoDB cluster set up. For this tutorial, you're going to create a [free M0 cluster](https://www.mongodb.com/basics/mongodb-atlas-tutorial). Once this is created, click "Load Sample Dataset." In the sample dataset, you'll see a database called `sample_restaurants` with a collection called `restaurants`, containing thousands of restaurants with reviews. This is the collection you'll focus on.

Now that you have your Azure Storage and MongoDB cluster setup, you are ready to deploy your federated database instance.

 1. Select "Data Federation" from the left-hand navigation menu.
 2. Click "Create New Federated Database" and, from the dropdown, select "Set up manually."
 3. Choose Azure as your cloud provider and give your federate database instance a name.
    ![The screen displaying our selection of cloud provider for our federated database, Azure.][1]
 4. To add your data source, click "Add Data Source" and select Azure Blob Storage as your data store.
 5. Next, you need to select an Azure Service Principal. You can use an existing one, but you'll create a new one for this tutorial. From the dropdown, select "Authorize an Azure Service Principal'' and click continue.
 6. To assign the relationships, follow the onscreen instructions. In this tutorial, you are going to use AzureCLI.
       1. First, you need to get the tenant ID by running the command: 
          ```bash
            az account show --query tenantId
            ```
       2. Next, run the following command to create your new Service Principle for Atlas, and copy the "id" from the output:
          ```bash
            az ad sp create --id 9efedfcc-2eca-4b27-a613-0cad1e114cb7
            ```
 7. You need to grant your access. Again, there will be prompts on the screen to guide you through the steps, but you can follow along here.
       1. First, run `az storage account list --query "[].id"` to get the Storage Account Resource ID. This will auto-populate the command on the page that you can copy and run in your terminal to set up the credentials delegation.
       2. After this, set up your storage container access. Choose whether you want to grant "Read-only" or "Read and write" privileges. Select your storage container region, and enter your storage container name. This will auto-populate the command below on the page that you can copy and paste in.
 8. Now, you need to provide sample pathways so you can query your data. If you've been following along with this simple example, copy the pathway `https://<YourStorageAccount>.blob.core.windows.net/<YourContainerName>/sample.json` — of course, with your account and container names — into the box.
 9. To link your Azure Blob Storage path components, from the dropdown, accept "any value (*)" and click “Next.”
 10. You should see your container name under your data sources now. All you need to do is drag your dataset and drop it into your virtual collection.

Let's add your MongoDB dataset to your federated database now.
 1. Click "Add Data Sources" and choose “Atlas Cluster.”
 2. Select the cluster that contains the sample dataset.
 3. This will bring up all the databases in the cluster. Click the dropdown arrow next to `sample_restaurants` to view the collections inside that database, choose `restaurants`, and click “Next.”
    ![The Add Data Source screen showing MongoDB Atlas Cluster selected, along with our collection restaurants.][2]
 4. Now, you should see `sample_restaurants.restaurants` under data sources. You can drag your dataset over to your virtual collection, just like before.

Now that you have both datasets in your federated database instance, click “Create,” and it's time to view your data.

### Connect to our federated database
There are many ways to view the data in the federated database. For this tutorial, you’ll create a simple Node.js application. To do this, you'll need a connection string. Once your federated database instance is created, you'll be able to see it under "Data Federation" on the left-hand navigation menu. On your instance, click “Connect.” Select “Driver” as your connection method and copy your connection string. Now, open up an IDE of your choice. 

For this tutorial, only a simple Node.js application is needed. If you want to learn more about developing with MongoDB and JavaScript, check out [Developer Center](https://www.mongodb.com/developer/languages/javascript/), where you’ll find a whole variety of tutorials, or explore MongoDB with other languages.

Before you start, make sure you have Node.js installed in your environment. 

 1. Set up a new Node.js project:
    - Create a new directory for your project.
    - Initialize a new Node.js project by running `npm init -y` in your terminal within that directory.
    - Install the MongoDB Node.js driver by running `npm install mongodb`.
 2. Create a JavaScript file:
    - Create a file named searchApp.js in your project directory.
 3. Implement the application:
     - Edit searchApp.js to include the following code, which connects to your MongoDB database and creates a client.
    ```
    const { MongoClient } = require('mongodb');
    
    // Connection URL
    const url = 'yourConnectionString';
    // Database Name
    const dbName = 'yourDatabaseName';
    // Collection Name
    const collectionName = 'yourCollectionName';

    // Create a new MongoClient
    const client = new MongoClient(url);
    ```
    - Now, create a function called `searchDatabase` that takes an input string and field from the command line and searches for documents containing that string in the specified field.
    ```
    // Function to search for a string in the database
    async function searchDatabase(fieldName, searchString) {
        try {
          await client.connect();
        console.log('Connected successfully to server');
          const db = client.db(dbName);
          const collection = db.collection(collectionName);
      
          // Dynamic query based on field name
          const query = { [fieldName]: { $regex: searchString, $options: "i" } };
          const foundDocuments = await collection.find(query).toArray();
          console.log('Found documents:', foundDocuments);
        } finally {
          await client.close();
        }
      }
    ```
    - Lastly, create a main function to control the flow of the application.
    ```
    // Main function to control the flow
    async function main() {
        // Input from command line arguments
        const fieldName = process.argv[2];
        const searchString = process.argv[3];

        if (!fieldName || !searchString) {
            console.log('Please provide both a field name and a search string as arguments.');
            return;
        }
    
        searchStringInDatabase(fieldName, searchString)
            .catch(console.error);
        }
      
    main().catch(console.error);
    ```
 4. Run your application with `node searchApp.js fieldName "searchString"`.
    - The script expects two command line arguments: the field name and the search string. It constructs a dynamic query object using these arguments, where the field name is determined by the first argument, and the search string is used to create a regex query.

In the terminal, you can type the query `node searchApp.js "Restaurant ID"  "40356030"` to find your `example2.csv` file as if it was stored in a MongoDB database. Or maybe `node searchApp.js borough "Manhattan"`, to find all restaurants in your virtual database (across all your databases) in Manhattan. You're not just limited to simple queries. Most operators and aggregations are available on your federated database. There are some limitations and variations in the MongoDB Operators and Aggregation Pipeline Stages on your federated database that you can read about in our [documentation](https://www.mongodb.com/docs/atlas/data-federation/supported-unsupported/supported-aggregation/#std-label-adf-aggregation-pipeline-support).

## Conclusion
By following the steps outlined, you've learned how to set up Azure Blob storage, upload diverse data formats like JSON and CSV, and connect these with your MongoDB dataset using a federated database. 

This tutorial highlights the potential of data federation in breaking down data silos, promoting data interoperability, and enhancing the overall data analysis experience. Whether you're a restaurant reviewer looking to share insights or a business seeking to unify disparate data sources, MongoDB's Data Federation along with Azure Blob storage provides a robust, scalable, and user-friendly platform to meet your data integration needs.

Are you ready to start building with Atlas on Azure? Get started for free today with [MongoDB Atlas on Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/mongodb.mongodb_atlas_self_serve_prod_2022?ocid=mdb_devrel_datafed_blobstorage_fy25). If you found this tutorial useful, make sure to check out some more of our articles in [Developer Center](https://www.mongodb.com/developer/), like [MongoDB Provider for EF Core Tutorial](https://www.mongodb.com/developer/languages/csharp/crud-changetracking-mongodb-provider-for-efcore/). Or pop over to our [Community Forums](https://www.mongodb.com/community/forums/) to see what other people in the community are building!

---


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt8526ba2a8dccdc22/65df43a9747141e57e0a356f/image2.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt12c4748f967ddede/65df43a837599950d070b53f/image1.png

A tutorial to guide you through integrating your Azure storage with MongoDB using Data Federation

Atlas Data Federation with Azure Blob Storage

Welcome to the exciting journey of transitioning from the traditional realm of relational databases to the dynamic world of MongoDB! This is the first entry in a series of tutorials helping you migrate from relational databases to MongoDB. Buckle up as we embark on a thrilling adventure filled with schema design, data modeling, and the wonders of the document model. Say goodbye to the rigid confines of tables and rows, and hello to the boundless possibilities of collections and documents. In this tutorial, we'll unravel the mysteries of MongoDB's schema design, exploring how to harness its flexibility to optimize your data storage like never before, using the Relational Migrator tool!

The migration from a relational database to MongoDB involves several stages. Once you've determined your database and application requirements, the initial step is schema design. This process involves multiple steps, all centered around how you intend to access your data. In MongoDB, data accessed together should be stored together. Let's delve into the schema design process.

## Schema design
The most fundamental difference between the world of relational databases and MongoDB is how your data is modeled. There are some terminology changes to keep in mind when moving from relational databases to MongoDB:

| RDBMS              | MongoDB                                          |
|--------------------|--------------------------------------------------|
| Database           | Database                                         |
| Table              | Collection                                       |
| Row                | Document                                         |
| Column             | Field                                            |
| Index              | Index                                            |
| JOIN               | Embedded document, document references, or $lookup to combine data from different collections |

Transitioning from a relational database to MongoDB offers several advantages due to the flexibility of JSON (JavaScript Object Notation) documents. [MongoDB's BSON](https://www.mongodb.com/docs/manual/reference/bson-types/) (Binary JSON) encoding extends JSON's capabilities by including additional data types like int, decimal, dates, and more, making it more efficient for representing complex data structures.

Documents in MongoDB, with features such as sub-documents (embedded documents) and arrays, align well with the structure of application-level objects. This alignment simplifies data mapping for developers, as opposed to the complexities of mapping object representations to tabular structures in relational databases, which can slow down development, especially when using Object Relational Mappers (ORMs).

When designing schemas for MongoDB, it's crucial to consider the application's requirements and leverage the document model's flexibility. While mirroring a relational database's flat schema in MongoDB might seem straightforward, it undermines the benefits of MongoDB's embedded data structures. For instance, MongoDB allows collapsing (embedding) data belonging to a parent-child relationship in relational databases into a single document, enhancing efficiency and performance. It's time to introduce a powerful tool that will streamline your transition from relational databases to MongoDB: the [Relational Migrator](https://www.mongodb.com/products/tools/relational-migrator). 

### Relational Migrator 
The transition from a relational database to MongoDB is made significantly smoother with the help of the Relational Migrator. The first step in this process is a comprehensive analysis of your existing relational schema. The Relational Migrator examines your database, identifying tables, relationships, keys, and other elements that define the structure and integrity of your data. You can connect to a live database or load a .SQL file containing Data Defining Language (DDL) statements. For this tutorial, I’m just going to use the sample schema available when you click **create new project**.

The first screen you’ll see is a diagram of your relational database relationships. This lays the groundwork by providing a clear picture of your current data model, which is instrumental in devising an effective migration strategy. By understanding the intricacies of your relational schema, the Relational Migrator can make informed suggestions on how to best transition this structure into MongoDB's document model.

![Relational database schema diagram][1]

Once your relational schema is analyzed, the Relational Migrator begins the task of suggesting MongoDB schemas. It offers suggestions for MongoDB schemas, translating relational concepts into document models. This isn't just about moving data around; it's about rethinking how that data can be structured to best utilize MongoDB's capabilities, such as its flexible document model and powerful query language.

Review the proposed document models, focusing on MongoDB's ability to handle embedded documents and references. Assess how these models align with your application's data access patterns and performance requirements. 

![MongoDB database schema diagram][2]

You now enter the tuning phase. It's here that you adjust these suggestions to tailor-fit your application. This stage is also about strategizing how best to leverage your data — considering aspects like access patterns, update frequencies, and the potential for data growth. For an in-depth guide on how to model your data in MongoDB, see our [data modeling guide](https://www.mongodb.com/basics/data-modeling).

In MongoDB, data that is accessed together should be stored together. This allows the avoidance of resource-intensive `$lookup` operations where not necessary. Evaluate whether to embed or reference data based on how it's accessed and updated. Remember, embedding can significantly speed up read operations but might complicate updates if the embedded data is voluminous or frequently changed. Use the Relational Migrator's suggestions as a starting point but remain flexible. Not every recommendation will be applicable, especially as you project data growth and access patterns into the future.

You may be stuck, staring at the daunting representation of your tables, wondering how to reduce this to a manageable number of collections that best meets your needs. Select any collection to see a list of suggestions for how to represent your data using embedded arrays or documents. Relational Migrator will show all the relationships in your database and how you can represent them in MongoDB, but they might not all be appropriate for application. In my example, I have selected the products collection.

![Suggested mappings for a collection][3]

Here, you can see I’m presented with a list of suggested mappings in the panel on the right side of the screen. A perfect suggestion for my use case is the categories. That is likely going to be read with the product document and will likely not be updated often, and presenting them together in one document will reduce the need for the `$lookup` operation in my application example. 

An example of a suggestion I may avoid would be orders. To look at this like an e-commerce app, it is reasonable to expect the orders array to grow indefinitely, as more and more people order this product. This unbounded array is considered an anti-pattern and will harm the database performance. Now that I’ve considered this, I can deselect the undesired fields and select **Apply**.

![Refined suggestions for a collection][4]

Even better, if I don’t ever access the supplier collection outside of the product collection, once I’ve embedded it, I can drop this collection, and simplify my database.

![New collection schema with embedded document][5]

If you want to see the relationships between two collections, select the link joining them. Here, you’ll see the Parent/Child relationship, the nature of the relationship (one-to-one/one-to-many/many-to-many), and the keys involved in linking the collections. Understanding the relationships in your data will help inform where you can make improvements in schema design.

![Relation between data in diagram][6]

Carefully consider the structure of each collection. The way you access and manipulate your data should guide the organization of your documents. The representation of relationships in MongoDB is a critical decision point. Embedding documents offers performance benefits for read-heavy operations, but references might be more suitable for datasets that are large or frequently updated. You can read more about this data concept in the [data modeling documentation](https://www.mongodb.com/docs/manual/data-modeling/concepts/embedding-vs-references/). Use the migrator’s suggestions to iteratively refine your new schema, and understand the suggestions are useful, but not all will make sense for you.

![Extremely long collection example][7]

While the tool provides valuable recommendations, it's crucial to adapt these to your application's specific context. There are limited cases where all your data in one doc will be an optimal decision for your migration. Your migration strategy should not be static. As you begin to work with your new MongoDB schema, it's essential to remain open to changes. Real-world application usage and performance metrics will guide further refinements, ensuring that your database architecture is not only optimized for current needs but also scalable for future demands.

Once you’re happy with your database design, select data migration from the top of the page. Here, just enter your old relational database connection string and your new MongoDB database connection string, and follow the on-screen instructions. If you want to learn more or get stuck in the process, the [Relational Migrator documentation](https://www.mongodb.com/docs/relational-migrator/connection-strings/mongodb-database-connection-strings/) contains all the information you need.

### Data modeling templates
It can be difficult to understand how best to store your data in your application, especially if you’re new to MongoDB. MongoDB Atlas offers a variety of data modeling templates that are designed to demonstrate best practices for various use cases. To find them, go to your project overview and you'll see the "Data Toolkit." Under this header, click the "Data Modeling Templates." These templates are there to serve as a good starting point to demonstrate best practices depending on how you plan on interacting with your data. 

![Data modelling templates.][8]

While you may not encounter an exact replica of your application, these templates serve to showcase optimal schema design practices based on access patterns. For most cases, something applicable or worth considering in your data model design will be presented, enabling you to maximize the effectiveness of the document model.

## Conclusion
Whether you're brand new to the MongoDB world or already familiar with the document model, migrating from a relational database to MongoDB requires careful consideration. These tools (such as Relational Migrator) and resources are there to speed up and inform your move to MongoDB and help you get the most out of your data. 

In our next tutorial, we will discuss querying, how to translate your SQL queries using Relational Migrator, and how to utilize indexes to minimize your write overhead, reduce your read times, and speed up your application as you move from relational databases to MongoDB.

If you found this tutorial useful, check out our article — [How to Model Your Documents for Vector Search](https://www.mongodb.com/developer/products/mongodb/doc-modeling-vector-search/) — or pop over to our [community forums](https://www.mongodb.com/community/forums/) to see what others are doing with MongoDB.


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7e10c8890fd54c23/65e8774877faff0e5a5a5cb8/image8.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt06b47b571be972b5/65e877485fd476466274f9ba/image5.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt6119b9e0d49180e0/65e877478b9c628cfa46feed/image1.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5542e55fba7a3659/65e8774863ec424da25d87e0/image4.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4e6f30cb08ace8c0/65e877478b9c62d66546fee9/image2.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt03ad163bd859e14c/65e877480395e457c2284cd8/image3.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1343d17dd36fd642/65e8774803e4602da8dc3870/image7.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt68bfa90356269858/65e87747105b937781a86cca/image6.png

Move from a relational database to MongoDB, and learn to use the document model.

Modernizing RDBMS Schemas With a MongoDB Document Model

Optimizing MongoDB performance involves understanding the intricacies of your database's schema and queries, and navigating this landscape might seem daunting. There can be a lot to keep in mind, but MongoDB Atlas provides several tools to help spot areas where how you interact with your data can be improved.

In this tutorial, we're going to go through what some of these tools are, where to find them, and how we can use what they tell us to get the most out of our database. Whether you're a DBA, developer, or just a MongoDB enthusiast, our goal is to empower you with the knowledge to harness the full potential of your data.

## Identify schema anti-patterns
As your application grows and use cases evolve, potential problems can present themselves in what was once a well-designed schema. How can you spot these? Well, in Atlas, from the data explorer screen, select the collection you'd like to examine. Above the displayed documents, you'll see a tab called "Schema Anti-Patterns." 

Now, in my collection, I have a board that describes the tasks necessary for our next sprint, so my documents look something like this: 

```json
{
  "boardName": "Project Alpha",
  "boardId": "board123",
  "tasks": [
    {
      "taskId": "task001",
      "title": "Design Phase",
      "description": "Complete the initial design drafts.",
      "status": "In Progress",
      "assignedTo": ["user123", "user456"],
      "dueDate": "2024-02-15",
    },
    // 10,000 more tasks
  ]
}
```

While this worked fine when our project was small in scope, the lists of tasks necessary really grew out of control (relatable, I'm sure). Let's pop over to our schema anti-pattern tab and see what it says.

![Collection schema anti-pattern page][1]

From here, you'll be provided with a list of anti-patterns detected in your database and some potential fixes. If we click the "Avoid using unbounded arrays in documents" item, we can learn a little more.

![Collection schema anti-pattern page, dropdown for more info.][2]

This collection has a few problems. Inside my documents, I have a substantial array. Large arrays can cause multiple issues, from exceeding the limit size on documents (16 MB) to degrading the performance of indexes as the arrays grow in size. Now that I have identified this, I can click "Learn How to Fix This Issue" to be taken to the [MongoDB documentation](https://www.mongodb.com/docs/atlas/schema-suggestions/avoid-unbounded-arrays/). In this case, the solution mentioned is [referencing](https://www.mongodb.com/docs/manual/data-modeling/#references). This involves storing the tasks in a separate collection and having a field to indicate what board they belong to. This will solve my issue of the unbounded array.

Now, every application is unique, and thus, how you use MongoDB to leverage your data will be equally unique. There is rarely one right answer for how to model your data with MongoDB, but with this tool, you are able to see what is slowing down your database and what you might consider changing — from [unused indexes](https://www.mongodb.com/docs/atlas/schema-suggestions/too-many-indexes/#remove-unnecessary-indexes) that are increasing your write operation times to over-reliance on the expensive `$lookup` operation, when [embedded documents](https://www.mongodb.com/docs/atlas/schema-suggestions/reduce-lookup-operations/) would do. 

## Performance Advisor
While you continue to use your MongoDB database, performance should always be at the back of your mind. Slow performance can hamper the user's experience with your application and can sometimes even make it unusable. With larger datasets and complex operations, these slow operations can become harder to avoid without conscious effort. The Performance Advisor provides a holistic view of your cluster, and as the name suggests, can help identify and solve the performance issues.

The Performance Advisor is a tool available for M10+ clusters and serverless instances. It monitors queries that MongoDB considers slow, based on how long operations on your cluster typically take. When you open up your cluster in MongoDB Atlas, you'll see a tab called "Performance Advisor."

![Performance Advisor landing page.][3]

This tool works by continuously monitoring query patterns and workload characteristics in real-time. Leveraging this data, the Performance Advisor automatically identifies potential performance issues and provides specific, actionable recommendations to optimize database operations.

These suggestions often include the creation of new indexes, modifications of existing ones, or the removal of redundant indexes, all aimed at reducing query execution time and improving overall efficiency. Without indexes, MongoDB has to scan all the documents in a collection to return a query result, but indexes allow MongoDB to limit the number of documents it must scan. Indexes improve your query performances, but they come at a performance cost for write operations. Because of this, a tool that can balance the pros and cons of each index becomes invaluable.

My database seems to be fine with the indexes I have at present and only spots a few areas of improvement with my schemas, but it is important to regularly check this page. That is because this page monitors a sample from the 20 most active collections. This means the performance advisor can regularly present new areas of optimization. Let's take a look at what one of these may look like.

From this example in the [MongoDB documentation](https://www.mongodb.com/docs/atlas/performance-advisor/index-ranking/), we have a database containing information on New York City taxi rides. A typical query on the application would look something like this:

```shell
db.yellow.find({ "dropoff_datetime": "2014-06-19 21:45:00",
                 "passenger_count": 1,
                 "trip_distance": {"$gt": 3 }
               })
```

With a large enough collection, running queries on specific field data will generate potentially slow operations without properly indexed collections. If we look at suggested indexes, we're presented with this screen, displaying the indexes we may want to create.

![Performance advisor suggested index page.][4]

We will be suggested potential indexes in order of performance impact, from most impactful to least. Impact is an indication of potential performance improvements the suggested index will bring. If we're happy with our suggestion, we can select "create index." If we want to refine the areas for recommendations to be shown, we can select specific collections from the dropdown or reduce the time range we want to analyze our cluster’s performance. 

## Query Profiler
While the MongoDB Performance Advisor excels at providing proactive, automated recommendations for indexing and general performance improvements, the MongoDB Query Profiler serves a distinct, complementary purpose. It's the go-to tool for a more in-depth and manual examination of your database's operations. 

The Query Profiler allows you to capture and analyze detailed information about individual queries, including execution time and resource utilization. This level of granularity is particularly useful when you need to diagnose and troubleshoot specific performance issues that are not directly related to indexing, such as slow-running queries or inefficient query patterns. With the Query Profiler, you can identify queries that are taking longer than expected, understand their execution plans, and pinpoint the exact stages in a query that are causing bottlenecks.

In the cluster dashboard, click on the “Profiler” tab. Atlas clusters have profiling enabled by default. On the Profiler tab, you will see a list of recent queries, commands, and operations. Each entry will display key information such as operation type, namespace, execution time, and more. Look for queries with long execution times. These are your slow queries.

![Performance advisor suggested index page.][5]

Click on any query to expand and view detailed information, including the query pattern, the number of documents returned, index usage, and execution statistics.

![Query profiler more details.][6]

Pay attention to queries that don’t use indexes or that scan a large number of documents. Consider creating new indexes or modifying existing ones to improve query efficiency.

After making changes like indexing, monitor the Query Profiler to observe any improvements in query performance. Regularly check the Query Profiler to stay on top of your database’s performance.

Remember, effective database management is an ongoing process, and regular monitoring and optimization are key to maintaining a healthy and efficient database system.

## Data modeling templates
We've seen how to examine our database, and the view to improve our data models and queries in our application, but the best method will always be starting with a good design. MongoDB Atlas offers a variety of data modeling templates that are designed to demonstrate best practices for various use cases. These templates serve as a valuable starting point, especially for those who are new to NoSQL databases or are looking to optimize their existing MongoDB schemas. To find them, go to your project overview and you'll see the "Data Toolkit." Under this header, click the "Data Modeling Templates."

These templates provide a host of examples demonstrating various use cases, from "Store Sales" to "Financial Analytics." These provide information on data modeling patterns, how best to query data stored like this, and lists of use cases in which these models can be helpful. Are you planning on storing unbounded data? Check out the "Customer Reviews" example. This demonstrates a good example of utilizing the [subset pattern](https://www.mongodb.com/blog/post/building-with-patterns-the-subset-pattern).

![Data modelling templates.][7]

It is important to consider that even though you might not see an exact example of what your application is, these templates are there to highlight best practices in schema design. You will likely see something that you can apply or should consider with how you are using your data, and learn how best to get the most out of the document model. 

## Conclusion
Optimizing your MongoDB database is an ongoing process. As your application evolves and your data grows, it's essential to regularly monitor and fine-tune your MongoDB Atlas deployment to ensure consistently high performance. 

In this tutorial, we've delved into the world of MongoDB Atlas, exploring the powerful tools and techniques available to optimize both schemas and queries. Stay proactive, monitor performance, and use the tools available to ensure your MongoDB Atlas databases run smoothly and efficiently.

If you would like to learn more about MongoDB and how best to design your data, pop over to the Developer Center and check out [A Summary of Schema Design Anti-Patterns and How to Spot Them](https://www.mongodb.com/developer/products/mongodb/schema-design-anti-pattern-summary/) or to our [Developer Community Forums](https://www.mongodb.com/community/forums/) to see what other people are building.


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta5008804d2f3ad0e/65b8cf0893cdf11de27cafc1/image3.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7fc9b26b8e7b5dcb/65b8cf077d4ae74bf4980919/image1.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcae2445ee8c4e5b1/65b8cf085f12eda542e220d7/image4.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3cddcbae41a303f4/65b8cf087d4ae7e2ee98091d/image5.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4ca5dd956a10c74c/65b8cf0830d47e0c7f5222f7/image7.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt07b872aa93e325e8/65b8cf0855a88a1fc1da7053/image6.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt64042035d566596c/65b8cf088fc5c08d430bcb76/image2.png

Learn how to get the most out of your MongoDB database using the tools provided to you by MongoDB Atlas.

Optimize With MongoDB Atlas: Performance Advisor, Query Analyzer, and More

From social media to streaming services, many applications require a mixture of different types of data. If you are designing an application that requires storing images or videos, a good idea is to store your media using a service specially designed to handle large objects of unstructured data. 

Your MongoDB database is not the best place to store your media files directly. The maximum BSON document size is 16MB. This helps ensure that a single document cannot use an excessive amount of RAM or, during transmission, an excessive amount of bandwidth. This provides an obstacle as this limit can easily be surpassed by images or videos. 

MongoDB provides [GridFS](https://www.mongodb.com/docs/manual/core/gridfs/) as a solution to this problem. MongoDB GridFS is a specification for storing and retrieving large files that exceed the BSON-document size limit and works by dividing the file into chunks and storing each chunk as a separate document. In a second collection, it stores the metadata for these files, including what chunks each file is composed of. While this may work for some use cases, oftentimes, it is a good idea to use a service dedicated to storing large media files and linking to that in your MongoDB document. Azure Blob (**B**inary **L**arge **Ob**jects) Storage is optimized for storing massive amounts of unstructured data and designed for use cases such as serving images directly to a browser, streaming video and audio, etc. Unstructured data is data that doesn't conform to a specific data model or format, such as binary data (how we store our media files).

In this tutorial, we are going to build a Java API with Spring Boot that allows you to upload your files, along with any metadata you wish to store. When you upload your file, such as an image or video, it will upload to [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs). It will store the metadata, along with a link to where the file is stored, in your MongoDB database. This way, you get all the benefits of MongoDB databases while taking advantage of how Azure Blob Storage deals with these large files.

![System architecture diagram for application][1]

Whether you're an experienced developer or just starting out, this guide is designed to equip you with the knowledge to build a media management back end that's not only scalable but also optimized for performance.

Let's dive into the prerequisites you'll need to follow this tutorial and then, step by step, we will build a system that is resilient, efficient, and ready to tackle the demands of a modern media-driven application.

## Prerequisites

 - [Java 11](https://www.java.com/en/download/manual.jsp) or higher
 - [Maven](https://maven.apache.org/) or Gradle, but this tutorial will reference Maven
 - A MongoDB cluster deployed and configured; if you need help, check out our [MongoDB Atlas tutorial](https://www.mongodb.com/basics/mongodb-atlas-tutorial) on how to get started
 - An [Azure account](https://azure.microsoft.com/en-us/free/) with an active subscription

## Set up Azure Storage

There are a couple of different ways you can set up your Azure storage, but we will use the Microsoft Azure Portal. Sign in with your [Azure account](https://portal.azure.com/#home) and it will take you to the home page. At the top of the page, search "Storage accounts."

![Microsoft portal dashboard showing Storage accounts option][2]

Select “Storage Accounts.” “Storage accounts (classic)” does not have all the features necessary for this tutorial.

This will take you to the page where you will create a storage account (or use an existing one). We are going to configure this storage account to be open and accessible to anyone who knows the URL. This is fine for this demo, but for production, you should configure this to only allow access from specific people/networks. Learn more about how to do this in the [Microsoft Azure documentation](https://learn.microsoft.com/en-us/azure/storage/common/authorize-data-access).

Select the subscription and resource group you wish to use, and give your storage account a name. The region, performance, and redundancy settings are depending on your plans with this application, but the lowest tiers have all the features we need.

In networking, select to enable public access from all networks. This might not be desirable for production but for following along with this tutorial, it allows us to bypass configuring rules for network access.

For everything else, we can accept the default settings. Once your storage account is created, we’re going to navigate to the resource. You can do this by clicking “Go to resource,” or return to the home page and it will be listed under your resources.

The next step is to set up a container. A container organizes a set of blobs, similar to a directory in a file system. A storage account can include an unlimited number of containers, and a container can store an unlimited number of blobs. On the left blade, select the containers tab, and click the plus container option. A menu will come up where you name your container (and configure access level if you don't want the default, private). Now, let's launch our container!

In order to connect your application to Azure Storage, create your `Shared Access Signature` (SAS). SAS allows you to have granular control over how your client can access the data. Select “Shared access signature” from the left blade menu and configure it to allow the services and resource types you wish to allow. For this tutorial, select “Object” under the allowed resource types. Object is for blob-level APIs, allowing operations on individual blobs, like uploading, downloading, or deleting an image. The rest of the settings you can leave as the default configuration. If you would like to learn more about what configurations are best suited for your application, check out [Microsoft’s documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview). Once you have configured it to your desired settings, click “Generate SAS and connection string.” Your SAS will be generated below this button.

![Microsoft portal storage account screen displaying the shared access signature page and where to locate the generate SAS and connection string][3]

## Create your API with Java and Spring Boot

Now that we have our Azure Storage account and MongoDB cluster set up, let's set up our API. What we're going to need is a way to upload files to our Blob Storage account and track the metadata for this file in a collection in our MongoDB database. 

### Getting started with Initializr

To set up our project, let's use the Spring Initializr. This will generate our pom.xml file, which will contain our dependencies for our project.

For this project, you want to select the options in the screenshot below and create a JAR:

![Spring initializer displaying the configuration for the application][4]

- Project: Maven
- Language: Java
- Dependencies: Spring Web, Spring Data MongoDB, and Azure Storage
- Generate: JAR

Open up the Maven project in the IDE of your choice and let's write some code!

### Code implementation

We are going to need some strings and keys to connect to Azure Blob Storage and our MongoDB database. To get your MongoDB connection string, log into [MongoDB Atlas](https://account.mongodb.com/account/login) and click “Connect.” If you need help, check out our [guide in the docs](https://www.mongodb.com/docs/guides/atlas/connection-string/).

![MongoDB Atlas dashboard showing where to find the connection string][5]

Add this to your `application.properties` file. Here we will also define the name of the database and the port we wish to run our application on. Don’t worry if the database or collection we use does not exist yet. The moment we send our data to it, MongoDB will create it.

```text
spring.data.mongodb.uri=mongodb+srv://<username>:<password>@<cluster>.mongodb.net/
spring.data.mongodb.database=userPosts
server.port=8080
```

Next, we need to add our Azure Blob Storage information. We are going to add the shared access signature and connection string we generated earlier and the name for our container.

```text
azure.blob-storage.connection-string=<connection.string>
spring.cloud.azure.storage.blob.container-name=<container.name>
```

Now that we have everything we need to get connected to our blob and database, we need to create an endpoint to upload an image stored locally and track the image metadata. First, we're going to create a model for our image metadata. Let's create a package in our project called `com.example.azureblob.model` and add our class `ImageMetadata.java`. 

```java
@Document("imageMetadata")
public class ImageMetadata {
	
	@Id
	private String id;
	private String caption;
	private String imageUrl;
	
	// getters and setters
}
```
> Tip: Most popular IDEs will have shortcuts to generate getters and setters.

At the start of our document, we use the annotation `@Document` to let Spring Boot know what collection in our MongoDB database this data will be added to. MongoDB will create this collection if it doesn't already exist. The `@Id` annotation indicates what will be used as the `_id`. If you don't assign an `_id`, MongoDB will automatically generate one for each new document. 

We added a couple of fields that describe our image being uploaded, including the URL it will be stored at. This URL will be generated when we insert our image into our blob storage. 

The next step is to use this model and interact with our data in the database. In Spring Boot, we use a repository interface. This has all the CRUD (create, read, update, delete) operations we will need. Let's create a package called `com.example.azureblob.repository` and add our interface `ImageMetadataRepository.java`.

```java
package com.example.azureblob.repositories;

import com.example.azureblob.models.ImageMetadata;
import org.springframework.data.mongodb.repository.MongoRepository;

public interface ImageMetadataRepository extends MongoRepository<ImageMetadata, String> {
}
```

Our repository will extend [MongoRepository](https://docs.spring.io/spring-data/mongodb/docs/current/api/org/springframework/data/mongodb/repository/MongoRepository.html). With MongoRepository, you don’t need to provide implementation code for many basic CRUD methods for your MongoDB database, such as save, findById, findAll, delete, etc. [Spring Data MongoDB](https://spring.io/projects/spring-data-mongodb) automatically generates the necessary implementation based on the method names and conventions.

Now that we have the repository set up, it's time to set up our service layer. This acts as the intermediate between our repository (data access layer) and our controller (REST endpoints) and contains the applications business logic. We'll create another package `com.example.azureblob.service` and add our class `ImageMetadataService.java`.

```java
@Service
public class ImageMetadataService {

    @Autowired
    private ImageMetadataRepository imageMetadataRepository;
    
    @Value("${spring.cloud.azure.storage.blob.container-name}")
    private String containerName;
    
    @Value("${azure.blob-storage.connection-string}")
    private String connectionString;

    private BlobServiceClient blobServiceClient;

    @PostConstruct
    public void init() {
        blobServiceClient = new BlobServiceClientBuilder().connectionString(connectionString).buildClient();
    }

    public ImageMetadata save(ImageMetadata metadata) {
        return imageMetadataRepository.save(metadata);
    }

    public List<ImageMetadata> findAll() {
        return imageMetadataRepository.findAll();
    }

    public Optional<ImageMetadata> findById(String id) {
        return imageMetadataRepository.findById(id);
    }
    
    public String uploadImageWithCaption(MultipartFile imageFile, String caption) throws IOException {
        String blobFileName = imageFile.getOriginalFilename();
        BlobClient blobClient = blobServiceClient.getBlobContainerClient(containerName).getBlobClient(blobFileName);

        blobClient.upload(imageFile.getInputStream(), imageFile.getSize(), true);

        String imageUrl = blobClient.getBlobUrl();
        
        ImageMetadata metadata = new ImageMetadata();
        metadata.setCaption(caption);
        metadata.setImageUrl(imageUrl);
        
        imageMetadataRepository.save(metadata);

        return "Image and metadata uploaded successfully!";
    }
}
```

Here we have a couple of our methods set up for finding our documents in the database and saving our metadata. Our `uploadImageWithCaption` method contains the integration with Azure Blob Storage. Here you can see we create a `BlobServiceClient` to interact with Azure Blob Storage. After it succeeds in uploading the image, it gets the URL of the uploaded blob. It then stores this, along with our other metadata for the image, in our MongoDB database.

Our last step is to set up a controller to establish our endpoints for the application. In a Spring Boot application, controllers handle requests, process data, and produce responses, making it possible to expose APIs and build web applications. Create a package `com.example.azureblob.service` and add the class `ImageMetadataController.java`.

```java
@RestController
@RequestMapping("/image-metadata")
public class ImageMetadataController {
	
	@Autowired
	private ImageMetadataService imageMetadataService;
	
	@PostMapping("/upload")
	public String uploadImageWithCaption(@RequestParam("image") MultipartFile imageFile, @RequestParam("caption") String caption) throws IOException {
		return imageMetadataService.uploadImageWithCaption(imageFile, caption);
	}
	
	@GetMapping("/")
	public List<ImageMetadata> getAllImageMetadata() {
		return imageMetadataService.findAll();
	}
	
	@GetMapping("/{id}")
	public ImageMetadata getImageMetadataById(@PathVariable String id) {
		return imageMetadataService.findById(id).orElse(null);
	}
}
```

Here we're able to retrieve all our metadata documents or search by `_id`, and we are able to upload our documents. 

This should be everything you need to upload your files and store the metadata in MongoDB. Let's test it out! You can use your favorite tool for testing APIs but I'll be using a cURL command. 

```console
curl -F "image=mongodb-is-webscale.png" -F "caption=MongoDB is Webscale" http://localhost:8080/blob/upload
```

Now, let's check how that looks in our database and Azure storage. If we look in our collection in MongoDB, we can see our metadata, including the URL to the image. Here we just have a few fields, but depending on your application, you might want to store information like when this document was created, the filetype of the data being stored, or even the size of the file.

![MongoDB Atlas showing our metadata document stored in the collection.][6]

If we want to see what this looks like in our blob, we can go back to our Azure portal and check our container. And here is our image!

![Microsoft portal showing our container and the image we transferred in.][7]

## Conclusion

As demonstrated in this tutorial, integrating Azure Blob Storage with a Java API powered by Spring Boot provides an optimal solution for storing large media files such as images and videos. Azure Blob Storage, Microsoft's cloud-based object storage solution, adeptly manages vast amounts of unstructured data. When coupled with MongoDB's efficient database handling, users benefit from a powerful tool that facilitates the uploading of files and storage of pertinent metadata.

Through this tutorial, we've walked you through setting up a MongoDB Atlas cluster and configuring Azure Storage. We demonstrated how to construct a Java API to seamlessly interact with both platforms. Furthermore, we detailed the essential steps, from the groundwork in the Spring Initializr to finalizing endpoints in our controller.

Whether you're aiming to build a sophisticated media platform, a content management system, or any application involving the storage of large media files, this tutorial gives you the steps to get started. By harnessing the combined prowess of Azure Blob Storage and MongoDB with a Spring Boot Java API, developers can build scalable, efficient, and resilient applications for the modern web.

Want to learn more about what you can do with Microsoft Azure and MongoDB? Check out some of our articles on [Developer Center](https://www.mongodb.com/developer/technologies/azure/), such as [How to Use Azure Functions with MongoDB Atlas in Java](https://www.mongodb.com/developer/products/atlas/azure-functions-mongodb-atlas-java/).


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb4993e491194e080/65783b3f2a3de32470d701d9/image3.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blte01b0191c2bca39b/65783b407cf4a90e91f5d32e/image5.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt64cc0729a4a237b9/65783b400d1fdc1b0b574582/image1.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf9ecb0e59cab32f5/65783b40bd48af73c3f67c16/image6.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltfb7a72157e335d34/65783b400f54458167a01f00/image4.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3117132d54aee8a1/65783b40fd77da8557159020/image7.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0807cbd5641b1011/65783b402813253a07cd197e/image2.png

This tutorial describes how to build a Spring Boot Application to upload your media files into Azure Blob Storage, while storing associated metadata in MongoDB.

Seamless Media Storage: Integrating Azure Blob Storage and MongoDB with Spring Boot

If your goal is to develop a multimedia platform, a robust content management system, or any type of application that requires storing substantial media files, the storage, retrieval, and management of these files are critical to delivering a seamless user experience. This is where a robust media management back end becomes an indispensable component of your tech stack. In this tutorial, we will guide you through the process of creating such a back end utilizing Node.js, Azure Blob Storage, and MongoDB.

Storing media files like images or videos directly in your MongoDB database may not be the most efficient approach. MongoDB has a BSON document size limit of 16MB, which is designed to prevent any single document from consuming too much RAM or bandwidth during transmission. Given the size of many media files, this limitation could be easily exceeded, presenting a significant challenge for storing large files directly in the database.

MongoDB's [GridFS](https://www.mongodb.com/docs/manual/core/gridfs/) is a solution for storing large files beyond the BSON-document size limit by dividing them into chunks and storing these chunks across separate documents. While GridFS is a viable solution for certain scenarios, an efficient approach is to use a dedicated service for storing large media files. Azure Blob (**B**inary **L**arge **Ob**jects) Storage, for example, is optimized for the storage of substantial amounts of unstructured data, which includes binary data like media files. Unstructured data refers to data that does not adhere to a specific model or format. 

We'll provide you with a blueprint to architect a backend system capable of handling large-scale media storage with ease, and show you how to post to it using cURL commands. By the end of this article, you'll have a clear understanding of how to leverage Azure Blob Storage for handling massive amounts of unstructured data and MongoDB for efficient data management, all orchestrated with a Node.js API that glues everything together.

![System architecture diagram for application.][1]

Whether you're an experienced developer or just starting out, this guide is designed to equip you with the knowledge to build a media management back end that's not only scalable but also optimized for performance with these large files.

First, we'll cover the necessary prerequisites for this tutorial. Following that, we'll proceed step by step to build the back end for an application that is robust, efficient, and equipped to handle the requirements of a modern, media-focused application.

## Prerequisites

 - **Node.js 18 or higher and npm**: Make sure you have [Node.js and npm (Node.js package manager)](https://nodejs.org/en/download) installed. Node.js is the runtime environment required to run your JavaScript code server-side. npm is used to manage the dependencies.
 - A MongoDB cluster deployed and configured. If you need help, check out our [MongoDB Atlas tutorial](https://www.mongodb.com/basics/mongodb-atlas-tutorial) on how to get started.
 - An [Azure account](https://azure.microsoft.com/en-us/free/) with an active subscription.

## Set up Azure Storage

For this tutorial, we will use the Microsoft Azure Portal to set up our Azure storage. Begin by logging into your [Azure account](https://portal.azure.com/#home) and it will take you to the home page. Once there, use the search bar at the top of the page to search "Storage accounts."

![Microsoft portal dashboard showing Storage accounts option][2]

Select “Storage Accounts.” Avoid selecting “Storage accounts (classic)” as this option does not have all the features required for this tutorial.

You'll be directed to a page where you can either create a new storage account or utilize an existing one. In this tutorial, we'll set up the storage account so that it's publicly accessible via its URL. While this setup is appropriate for a demonstration, in a production environment, it's crucial to restrict access to designated individuals or networks for security purposes. Detailed instructions on configuring these access controls can be found in the [Microsoft Azure documentation](https://learn.microsoft.com/en-us/azure/storage/common/authorize-data-access).

Choose your preferred subscription and resource group, then assign a name to your storage account. While the selection of region, performance, and redundancy options will vary based on your application's requirements, the basic tiers will suffice for all the functionalities required in this tutorial.

In the networking section, opt to allow public access from all networks. While this setting is generally not recommended for production environments, it simplifies the process for this tutorial by eliminating the need to set up specific network access rules.

For the rest of the configuration settings, we can accept the default settings. Once your storage account is created, we’re going to navigate to the resource. You can do this by clicking “Go to resource,” or return to the home page and it will be listed under your resources.

Now, we'll proceed to create a container. Think of a container as akin to a directory in a file system, used for organizing blobs. You can have as many containers as you need in a storage account, and each container can hold numerous blobs. To do this, go to the left panel and click on the Containers tab, then choose the “plus container” option. This will open a dialog where you can name your container and, if necessary, alter the access level from the default private setting. Once that's done, you can go ahead and initiate your container.

To connect your application to Azure Storage, you'll need to create a `Shared Access Signature` (SAS). SAS provides detailed control over the ways your client can access data. From the menu on the left, select “Shared access signature” and set it up to permit the services and resource types you need. For the purposes of this tutorial, choose “Object” under allowed resource types, which is suitable for blob-level APIs and enables operations on individual blobs, such as upload, download, or delete.

You can leave the other settings at their default values. However, if you're interested in understanding which configurations are ideal for your application, [Microsoft’s documentation](https://learn.microsoft.com/en-us/azure/storage/common/storage-sas-overview) offers comprehensive guidance. Once you've finalized your settings, click “Generate SAS and connection string.” This action will produce your SAS, displayed below the button.

![Microsoft portal storage account screen displaying the shared access signature page and where to locate the generate SAS and connection string][3]

Create a .env file in the same location as you’ll be creating your application to store your environment variables. Add in your SAS token here. We’ll also add our account name for our storage account, and the name of our container. Your account name is displayed at the top of the overview page, and the container name can be found by clicking “Container” on the left blade menu. Here you’ll get a list of containers available to use.

```console
SAS_TOKEN="YOUR_SAS_TOKEN"
ACCOUNT_NAME="YOUR_ACCOUNT_NAME"
CONTAINER_NAME="YOUR_CONTAINER_NAME"
```

## Connecting our application 

Alright, let's get things rolling with connecting to our Azure blob storage and MongoDB database. To get your MongoDB connection string, log into [MongoDB Atlas](https://account.mongodb.com/account/login) and click connect. If you need help, check out our [guide in the docs](https://www.mongodb.com/docs/guides/atlas/connection-string/).

![MongoDB Atlas dashboard showing where to find the connection string.][4]

Copy this string and add it to the .env file with the name `MONGODB_URI`.

```console
MONGODB_URI="mongodb+srv://<user>:<password>@<cluster>.mongodb.net/"
```

## Downloading your packages
To initialize our project, we’re going to use the following command:

```console
npm install --save @azure/storage-blob mongodb dotenv
```

This command will install the `@azure/storage-blob` package for interacting with Azure Blob Storage, `mongodb` for MongoDB operations, and `dotenv` for loading environment variables.

Let’s import our dependencies. At the top of our `app.mjs` file, we’ll add the following lines. 

```javascript
import http from 'http';
import { BlobServiceClient } from '@azure/storage-blob';
import { MongoClient } from 'mongodb';
import 'dotenv/config';
```

Below this, we can configure the environment variables we will be using, and connect to our Azure blob storage and MongoDB database.

```javascript
// Load in environment variables
const mongodbUri = process.env.MONGODB_URI;
const accountName = process.env.ACCOUNT_NAME;
const sasToken = process.env.SAS_TOKEN;
const containerName = process.env.CONTAINER_NAME;

// Establishes a connection with Azure Blob Storage
const blobServiceClient = new BlobServiceClient(`https://${accountName}.blob.core.windows.net/?${sasToken}`);
const containerClient = blobServiceClient.getContainerClient(containerName);

// Connect to MongoDB
const client = new MongoClient(mongodbUri);
client.connect();

```

## Uploading your file

We need to establish somewhere to make our API calls. We are going to create a new HTTP server. The http.createServer() method is part of [Node.js's HTTP module](https://nodejs.org/api/http.html) and takes a request listener function as an argument. In this case, `handleImageUpload` is passed as the request listener, which means that this function will be called every time the server receives an HTTP request.

```js
const server = http.createServer(handleImageUpload);
const port = 3000;
server.listen(port, () => {
  console.log(`Server listening on port ${port}`);
});
```

The `handleImageUpload` function is designed to process HTTP POST requests to the /api/upload endpoint, handling the uploading of an image and the storing of its associated metadata. It will call upon a couple of helper functions to achieve this. We’ll break down how these work as well.

```javascript
async function handleImageUpload(req, res) {
  res.setHeader('Content-Type', 'application/json');
  if (req.url === '/api/upload' && req.method === 'POST') {
    try {
      // Extract metadata from headers
      const {fileName, caption, fileType } = await extractMetadata(req.headers);

      // Upload the image as a to Azure Storage Blob as a stream
      const imageUrl = await uploadImageStreamed(fileName, req);

      // Store the metadata in MongoDB
      await storeMetadata(fileName, caption, fileType, imageUrl);

      res.writeHead(201);
      res.end(JSON.stringify({ message: 'Image uploaded and metadata stored successfully', imageUrl }));
    } catch (error) {
      console.error('Error:', error);
      res.writeHead(500);
      res.end(JSON.stringify({ error: 'Internal Server Error' }));
    }
  } else {
    res.writeHead(404);
    res.end(JSON.stringify({ error: 'Not Found' }));
  }
}
```

If the incoming request is a POST to the correct endpoint, it will call our `extractMetadata` method. This function takes in our header from the request and extracts the associated metadata. 

```javascript
async function extractMetadata(headers) {
  const contentType = headers['content-type'];
  const fileType = contentType.split('/')[1];
  const contentDisposition = headers['content-disposition'] || '';
  const caption = headers['x-image-caption'] || 'No caption provided';
  const matches = /filename="([^"]+)"/i.exec(contentDisposition);
  const fileName = matches?.[1] || `image-${Date.now()}.${fileType}`;
  return { fileName, caption, fileType };
}
```
It assumes that the 'content-type' header of the request will include the file type (like image/png or image/jpeg). It extracts this file type from the header. It then attempts to extract a filename from the content-disposition header, if provided. If no filename is given, it generates a default one using a timestamp.

Using the extracted or generated filename and file type, along with the rest of our metadata from the header, it calls `uploadImageStreamed`, which uploads the image as a stream directly from the request to Azure Blob Storage.

```javascript
async function uploadImageStreamed(blobName, dataStream) {
  const blobClient = containerClient.getBlockBlobClient(blobName);
  await blobClient.uploadStream(dataStream);
  return blobClient.url;
}
```

In this method, we are creating our `blobClient`. The blobClient opens a connection to an Azure Storage blob and allows us to manipulate it. Here we upload our stream into our blob and finally return our blob URL to be stored in MongoDB. 

Once we have our image stored in Azure Blob Storage, we are going to take the URL and store it in our database. The metadata you decide to store will depend on your application. In this example, I add a caption for the file, the name, and the URL, but you might also want information like who uploaded the image or when it was uploaded. This document is inserted into a MongoDB collection using the `storeMetadata` method.

```javascript
async function storeMetadata(name, caption, fileType, imageUrl) {
  const collection = client.db("tutorial").collection('metadata');
  await collection.insertOne({ name, caption, fileType, imageUrl });
}
```

Here we create and connect to our MongoClient, and insert our document into the metadata collection in the tutorial. Don’t worry if the database or collection don’t exist yet. As soon as you try to insert data, MongoDB will create it.

If the upload and metadata storage are successful, it sends back an HTTP 201 status code and a JSON response confirming the successful upload.

Now we have an API call to upload our image, along with some metadata for said image. Let's test what we built! Run your application by executing the `node app.mjs` command in a terminal that's open in your app's directory. If you’re following along, you’re going to want to substitute the path to the image below to your own path, and whatever you want the metadata to be.

```console
curl -X POST \
  -H "Content-Type: image/png" \
  -H "Content-Disposition: attachment; filename=\"mongodb-is-webscale.png\"" \
  -H "X-Image-Caption: Your Image Caption Here" \
  --data-binary @"/path/to/your/mongodb-is-webscale.png" \
  http://localhost:3000/api/upload
```

There’s a couple of steps to our cURL command. 
 - `curl -X POST` initiates a curl request using the POST method, which is commonly used for submitting data to be processed to a specified resource.
 - `-H "Content-Type: image/png"` includes a header in the request that tells the server what the type of the content being sent is. In this case, it indicates that the file being uploaded is a PNG image.
 - `-H "Content-Disposition: attachment; filename=\"mongodb-is-webscale.png\""` header is used to specify information about the file. It tells the server the file should be treated as an attachment, meaning it should be downloaded or saved rather than displayed. The filename parameter is used to suggest a default filename to be used if the content is saved to a file. (Otherwise, our application will auto-generate one.) 
 - `-H "X-Image-Caption: Your Image Caption Here"` header is used to dictate our caption. Following the colon, include the message you wish to store in or MongoDB document.
 - `--data-binary @"{Your-Path}/mongodb-is-webscale.png"` tells cURL to read data from a file and to preserve the binary format of the file data. The @ symbol is used to specify that what follows is a file name from which to read the data. {Your-Path} should be replaced with the actual path to the image file you're uploading.
 - `http://localhost:3000/api/upload` is the URL where the request is being sent. It indicates that the server is running on localhost (the same machine from which the command is being run) on port 3000, and the specific API endpoint handling the upload is /api/upload.

Let’s see what this looks like in our storage. First, let's check our Azure Storage blob. You can view the `mongodb-is-webscale.png` image by accessing the container we created earlier. It confirms that the image has been successfully stored with the designated name.

![Microsoft portal showing our container and the image we transferred in.][5]

Now, how can we retrieve this image in our application? Let’s check our MongoDB database. You can do this through the MongoDB Atlas UI. Select the cluster and the collection you uploaded your metadata to. Here you can view your document.

![MongoDB Atlas showing our metadata document stored in the collection.][6]

You can see we’ve successfully stored our metadata! If you follow the URL, you will be taken to the image you uploaded, stored in your blob. 

## Conclusion

Integrating Azure Blob Storage with MongoDB provides an optimal solution for storing large media files, such as images and videos, and provides a solid backbone for building your multimedia applications. Azure Blob Storage, a cloud-based service from Microsoft, excels in handling large quantities of unstructured data. This, combined with the efficient database management of MongoDB, creates a robust system. It not only simplifies the file upload process but also effectively manages relevant metadata, offering a comprehensive solution for data storage needs.

Through this tutorial, we've provided you with the steps to set up a  MongoDB Atlas cluster and configure Azure Storage, and we demonstrated how to construct a Node.js API to seamlessly interact with both platforms.

If your goal is to develop a multimedia platform, a robust content management system, or any type of application that requires storing substantial media files, this guide offers a clear pathway to embark on that journey. Utilizing the powerful capabilities of Azure Blob Storage and MongoDB, along with a Node.js API, developers have the tools to create applications that are not only scalable and proficient but also robust enough to meet the demands of today's dynamic web environment.

Want to learn more about what you can do with Microsoft Azure and MongoDB? Check out some of our articles in [Developer Center](https://www.mongodb.com/developer/technologies/azure/), such as [Building a Crypto News Website in C# Using the Microsoft Azure App Service and MongoDB Atlas](https://www.mongodb.com/developer/languages/csharp/crypto-news-website/), where you can learn how to build and deploy a website in just a few simple steps.


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd9281432bdaca405/65797bf8177bfa1148f89ad7/image3.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc26fafc9dc5d0ee6/65797bf87cf4a95dedf5d9cf/image2.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltedfcb0b696b631af/65797bf82a3de30dcad708d1/image4.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt85d7416dc4785d29/65797bf856ca8605bfd9c50e/image5.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1c7d6af67a124be6/65797bf97ed7db1ef5c7da2f/image6.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt6b50b6db724830a1/65797bf812bfab1ac0bc3a31/image1.png

Learn to create your own media management backend, storing your media files in Azure Blob Storage, and associated metadata in MongoDB.

Building a Scalable Media Management Back End: Integrating Node.js, Azure Blob Storage, and MongoDB

Anaiya Raisinghani

Associate Developer Advocate

Prakul Agarwal

We are excited to announce native support for [MongoDB Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) in [Microsoft Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/overview/). With this integration, users can bring the power of LLMs (large language models) to their proprietary data securely, and build generative AI applications using RAG (retrieval-augmented generation) with programming languages like Python and C#. The accompanying tutorial will walk you through an example.

## What is Semantic Kernel?
Semantic Kernel is a polyglot, open-source SDK that lets users combine various AI services with their applications. Semantic Kernel uses connectors to allow you to swap out AI services without rewriting code. Components of Semantic Kernel include: 

 - AI services: Supports AI services like OpenAI, Azure OpenAI, and Hugging Face. 
 - Programming languages: Supports conventional programming languages like C# Python, and Java.
 - Large language model (LLM) prompts: Supports the latest in LLM AI prompts with prompt templating, chaining, and planning capabilities. 
 - Memory: Provides different vectorized stores for storing data, including MongoDB.

## What is MongoDB Atlas Vector Search?
MongoDB Atlas Vector Search is a fully managed service that simplifies the process of effectively indexing high-dimensional vector embedding data within MongoDB and being able to perform fast vector similarity searches. 

Embedding refers to the representation of words, phrases, or other entities as dense vectors in a continuous vector space. It's designed to ensure that words with similar meanings are grouped closer together. This method helps computer models better understand and process language by recognizing patterns and relationships between words and is what allows us to search by semantic meaning.

When data is converted into numeric vector embeddings using encoding models, these embeddings can be stored directly alongside their respective source data within the MongoDB database. This co-location of vector embeddings and the original data not only enhances the efficiency of queries but also eliminates potential synchronization issues. By avoiding the need to maintain separate databases or synchronization processes for the source data and its embeddings, MongoDB provides a seamless and integrated data retrieval experience.

This consolidated approach streamlines database management and allows for intuitive and sophisticated semantic searches, making the integration of AI-powered experiences easier.

## Microsoft Semantic Kernel and MongoDB
This combination enables developers to build AI-powered intelligent applications using MongoDB Atlas Vector Search and large language models from providers like OpenAI, Azure OpenAI, and Hugging Face. 

Despite all their incredible capabilities, LLMs have a knowledge cutoff date and often need to be augmented with proprietary, up-to-date information for the particular business that an application is being built for. This “long-term memory for LLM” capability for AI-powered intelligent applications is typically powered by leveraging [vector embeddings](https://www.mongodb.com/developer/products/atlas/building-generative-ai-applications-vector-search-open-source-models/). Semantic Kernel allows for storing and retrieving this vector context for AI apps using the memory plugin (which now has support for MongoDB Atlas Vector Search). 

## Tutorial
[Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) is integrated in this tutorial to provide a way to interact with our memory store that was created through our MongoDB and Semantic Kernel [connector](https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/connectors/memory/mongodb_atlas).

This tutorial takes you through how to use Microsoft Semantic Kernel to properly upload and embed documents into your MongoDB Atlas cluster, and then conduct queries using Microsoft Semantic Kernel as well, all in Python!

## Pre-requisites 

 - [MongoDB Atlas cluster](https://www.mongodb.com/docs/atlas/getting-started/) 
 - IDE of your choice (this tutorial uses [Google Colab](https://colab.research.google.com/drive/1Qlgor8-nFEkugMdTADYVKIYZejSAWQUV#scrollTo=U8u9vfQYnWvt) — please refer to it if you’d like to run the commands directly)
 - [OpenAI API key](https://platform.openai.com/docs/introduction)

Let’s get started!

## Setting up our Atlas cluster
Visit the [MongoDB Atlas](https://account.mongodb.com/account/login) dashboard and set up your cluster. In order to take advantage of the `$vectorSearch` operator in an aggregation pipeline, you need to run MongoDB Atlas 6.0.11 or higher. This tutorial can be built using a free cluster. 

When you’re setting up your deployment, you’ll be prompted to set up a database user and rules for your network connection. Please ensure you save your username and password somewhere safe and have the correct IP address rules in place so your cluster can connect properly. 

If you need more help getting started, check out our [tutorial on MongoDB Atlas](https://www.mongodb.com/basics/mongodb-atlas-tutorial). 

## Installing the latest version of Semantic Kernel
In order to be successful with our tutorial, let’s ensure we have the most up-to-date version of Semantic Kernel installed in our IDE. As of the creation of this tutorial, the latest version is 0.3.14. Please run this `pip` command in your IDE to get started:
```
!python -m pip install semantic-kernel==0.3.14.dev
```
Once it has been successfully run, you will see various packages being downloaded. Please ensure `pymongo` is downloaded in this list. 

## Setting up our imports 
Here, include the information about our OpenAI API key and our connection string. 

Let’s set up the necessary imports:
```
import openai
import semantic_kernel as sk
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion, OpenAITextEmbedding
from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasMemoryStore



kernel = sk.Kernel()


openai.api_key = '<your-openai-key'
MONGODB_SEARCH_INDEX="defaultRandomFacts"
MONGODB_DATABASE="semantic-kernel"
MONGODB_COLLECTION="randomFacts"


kernel.add_chat_service("chat-gpt", OpenAIChatCompletion("gpt-3.5-turbo", openai.api_key))
kernel.add_text_embedding_generation_service("ada", OpenAITextEmbedding("text-embedding-ada-002", openai.api_key))


kernel.register_memory_store(memory_store=MongoDBAtlasMemoryStore(index_name=MONGODB_SEARCH_INDEX, database_name=MONGODB_DATABASE, connection_string="<mongodb-uri>"))
kernel.import_skill(sk.core_skills.TextMemorySkill())
```
Importing in OpenAI is crucial because we are using their data model to embed not only our documents but also our queries. We also want to import their Text Embedding library for this same reason. For this tutorial, we are using the embedding model `ada-002`, but please double check that you’re using a model that is compatible with your OpenAI API key.

Our `MongoDBAtlasMemoryStore` class is very important as it’s the part that enables us to use MongoDB as our memory store. This means we can connect to the Semantic Kernel and have our documents properly saved and formatted in our cluster. For more information on this class, please refer to the [repository](https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/connectors/memory/mongodb_atlas). 

This is also where you will need to incorporate your OpenAI API key along with your [MongoDB connection string](https://www.mongodb.com/docs/atlas/), and other important variables that we will use. The ones above are just a suggestion, but if they are changed while attempting the tutorial, please ensure they are consistent throughout. For help on accessing your OpenAI key, please read the section below.

### Generate your OpenAI key
In order to generate our embeddings, we will use the OpenAI API. First, we’ll need a secret key. To create your OpenAI key, you'll need to create an account. Once you have that, visit the [OpenAI API](https://platform.openai.com/) and you should be greeted with a screen like the one below. Click on your profile icon in the top right of the screen to get the dropdown menu and select “View API keys”.
![where to find your API key on OpenAI](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltbb8293d6a59ed869/65568a482e0ea1959117960a/semantickernela1.png)

Here, you can generate your own API key by clicking the “Create new secret key” button. Give it a name and store it somewhere safe. This is all you need from OpenAI to use their API to generate your embeddings.

![how to create a new API key](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1550895c161b4e43/65568a9b4452cc73ecf967a6/semantickernela2.png)

## The need for retrieval-augmented generation (RAG)
Retrieval-augmented regeneration, also known as RAG, is an NLP technique that can help improve the quality of large language models (LLMs). It’s an artificial intelligence framework for getting data from an external knowledge source. The memory store we are creating using Microsoft Semantic Kernel is an example of this. But why is RAG necessary? Let’s take a look at an example.

LLMs like OpenAI GPT-3.5 exhibit an impressive and wide range of skills. They are trained on the data available on the internet about a wide range of topics and can answer queries accurately. Using Semantic Kernel, let’s ask OpenAI’s LLM if Albert Einstein likes coffee: 
```
# Wrap your prompt in a function
prompt = kernel.create_semantic_function("""
As a friendly AI Copilot, answer the question: Did Albert Einstein like coffee?
""")


print(prompt())
```
The output received is:
```
Yes, Albert Einstein was known to enjoy coffee. He was often seen with a cup of coffee in his hand and would frequently visit cafes to discuss scientific ideas with his colleagues over a cup of coffee.
```
Since this information was available on the public internet, the LLM was able to provide the correct answer.

But LLMs have their limitations: They have a knowledge cutoff (September 2021, in the case of OpenAI) and do not know about proprietary and personal data. They also have a tendency to hallucinate — that is, they may confidently make up facts and provide answers that may seem to be accurate but are actually incorrect. Here is an example to demonstrate this knowledge gap:

```
prompt = kernel.create_semantic_function("""
As a friendly AI Copilot, answer the question: Did I like coffee?
""")


print(prompt())
```
The output received is:
```
As an AI, I don't have personal preferences or experiences, so I can't say whether "I" liked coffee or not. However, coffee is a popular beverage enjoyed by many people around the world. It has a distinct taste and aroma that some people find appealing, while others may not enjoy it as much. Ultimately, whether someone likes coffee or not is a subjective matter and varies from person to person.
``` 

As you can see, there is a knowledge gap here because we don’t have our personal data loaded in OpenAI that our query can access. So let’s change that. Continue on through the tutorial to learn how to augment the knowledge base of the LLM with proprietary data. 

## Add some documents into our MongoDB cluster
Once we have incorporated our MongoDB connection string and our OpenAI API key, we are ready to add some documents into our MongoDB cluster. 

Please ensure you’re specifying the proper collection variable below that we set up above. 
```
async def populate_memory(kernel: sk.Kernel) -> None:
# Add some documents to the semantic memory
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="1", text="We enjoy coffee and Starbucks"
)
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="2", text="We are Associate Developer Advocates at MongoDB"
)
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="3", text="We have great coworkers and we love our teams!"
)
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="4", text="Our names are Anaiya and Tim"
)
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="5", text="We have been to New York City and Dublin"
)
```
Here, we are using the `populate_memory` function to define five documents with various facts about Anaiya and Tim. As you can see, the name of our collection is called “randomFacts”, we have specified the ID for each document (please ensure each ID is unique, otherwise you will get an error), and then we have included a text phrase we want to embed. 

Once you have successfully filled in your information and have run this command, let’s add them to our cluster — aka let’s populate our memory! To do this, please run the command:
```
print("Populating memory...aka adding in documents")
await populate_memory(kernel)
```
Once this command has been successfully run, you should see the database, collection, documents, and their embeddings populate in your Atlas cluster. The screenshot below shows how the first document looks after running these commands.

![screenshot of our first document with the text embedding](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0a505f52e4d85526/65568b25a255e23bc9211f7a/semantickernela3.png)
  
Once the documents added to our memory have their embeddings, let’s set up our search index and ensure we can generate embeddings for our queries. 

## Create a vector search index in MongoDB 
In order to use the `$vectorSearch` operator on our data, we need to set up an appropriate search index. We’ll do this in the [Atlas UI](https://account.mongodb.com/account/login). Select the “Search" tab on your cluster and click “Create Search Index”.
![where to create a vector search index](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4fb2a898919fdbc2/65568b844e0195c4c3bf1f64/semantickernela4.png)

We want to choose the "JSON Editor Option" and click "Next".

![specifics for creating a search index](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt33055bae582002e4/655bb51b3133dbdcc8911770/semantickernel5.png)

On this page, we're going to select our target database, `semantic-kernel`, and collection, `randomFacts`.

For this tutorial, we are naming our index `defaultRandomFacts`. The index will look like this: 

```json
{
  "mappings": {
    "dynamic": true,
    "fields": {
      "embedding": {
        "dimensions": 1536,
        "similarity": "dotProduct",
        "type": "knnVector"
      }
    }
  }
}
```
The fields specify the embedding field name in our documents, `embedding`, the dimensions of the model used to embed, `1536`, and the similarity function to use to find K-nearest neighbors, `dotProduct`. It's very important that the dimensions in the index match that of the model used for embedding. This data has been embedded using the same model as the one we'll be using, but other models are available and may use different dimensions.

Check out our [Vector Search documentation](https://www.mongodb.com/docs/atlas/atlas-search/vector-search/) for more information on the index configuration settings.

## Query documents using Microsoft Semantic Kernel
In order to query your new documents hosted in your MongoDB cluster “memory” store, we can use the `memory.search_async` function. Run the following commands and watch the magic happen:

```
result = await kernel.memory.search_async(MONGODB_COLLECTION, 'What is my job title?')


print(f"Retrieved document: {result[0].text}, {result[0].relevance}")
```

Now you can ask any question and get an accurate response!  

Examples of questions asked and the results: 
![the result of the question: What is my job title?](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb0ffef5ac86f7f5c/655bb5826f7bf46d9a82c7fe/semantickernel6.png)

![the result of the question: Do I like caffeine?](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4dcf417349dfc947/655bb5b4867c0b640138726c/semantickernel7.png)

![the result of the question: Where in the world have I visited?](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3f87d869993ff195/655bb5e17c56dde9cba5dcac/semantickernel8.png)

## Conclusion
In this tutorial, you have learned a lot of very useful concepts:

 - What Microsoft Semantic Kernel is and why it’s important.
 - How to connect Microsoft Semantic Kernel to a MongoDB Atlas cluster.
 - How to add in documents to your MongoDB memory store (and embed them, in the process, through Microsoft Semantic Kernel).
 - How to query your new documents in your memory store using Microsoft Semantic Kernel.

For more information on MongoDB Vector Search, please visit the [documentation](https://www.mongodb.com/products/platform/atlas-vector-search), and for more information on Microsoft Semantic Kernel, please visit their [repository](https://github.com/microsoft/semantic-kernel) and [resources](https://learn.microsoft.com/en-us/semantic-kernel/overview/). 

If you have any questions, please visit our [MongoDB Developer Community Forum](https://www.mongodb.com/community/forums/). 

Follow this comprehensive guide to getting started with Microsoft Semantic Kernel and MongoDB Atlas Vector Search.