Anaiya Raisinghani

Prakul Agarwal

Tim Kelly

We are excited to announce native support for [MongoDB Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) in [Microsoft Semantic Kernel](https://learn.microsoft.com/en-us/semantic-kernel/overview/). With this integration, users can bring the power of LLMs (large language models) to their proprietary data securely, and build generative AI applications using RAG (retrieval-augmented generation) with programming languages like Python and C#. The accompanying tutorial will walk you through an example.

## What is Semantic Kernel?
Semantic Kernel is a polyglot, open-source SDK that lets users combine various AI services with their applications. Semantic Kernel uses connectors to allow you to swap out AI services without rewriting code. Components of Semantic Kernel include: 

 - AI services: Supports AI services like OpenAI, Azure OpenAI, and Hugging Face. 
 - Programming languages: Supports conventional programming languages like C# Python, and Java.
 - Large language model (LLM) prompts: Supports the latest in LLM AI prompts with prompt templating, chaining, and planning capabilities. 
 - Memory: Provides different vectorized stores for storing data, including MongoDB.

## What is MongoDB Atlas Vector Search?
MongoDB Atlas Vector Search is a fully managed service that simplifies the process of effectively indexing high-dimensional vector embedding data within MongoDB and being able to perform fast vector similarity searches. 

Embedding refers to the representation of words, phrases, or other entities as dense vectors in a continuous vector space. It's designed to ensure that words with similar meanings are grouped closer together. This method helps computer models better understand and process language by recognizing patterns and relationships between words and is what allows us to search by semantic meaning.

When data is converted into numeric vector embeddings using encoding models, these embeddings can be stored directly alongside their respective source data within the MongoDB database. This co-location of vector embeddings and the original data not only enhances the efficiency of queries but also eliminates potential synchronization issues. By avoiding the need to maintain separate databases or synchronization processes for the source data and its embeddings, MongoDB provides a seamless and integrated data retrieval experience.

This consolidated approach streamlines database management and allows for intuitive and sophisticated semantic searches, making the integration of AI-powered experiences easier.

## Microsoft Semantic Kernel and MongoDB
This combination enables developers to build AI-powered intelligent applications using MongoDB Atlas Vector Search and large language models from providers like OpenAI, Azure OpenAI, and Hugging Face. 

Despite all their incredible capabilities, LLMs have a knowledge cutoff date and often need to be augmented with proprietary, up-to-date information for the particular business that an application is being built for. This “long-term memory for LLM” capability for AI-powered intelligent applications is typically powered by leveraging [vector embeddings](https://www.mongodb.com/developer/products/atlas/building-generative-ai-applications-vector-search-open-source-models/). Semantic Kernel allows for storing and retrieving this vector context for AI apps using the memory plugin (which now has support for MongoDB Atlas Vector Search). 

## Tutorial
[Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) is integrated in this tutorial to provide a way to interact with our memory store that was created through our MongoDB and Semantic Kernel [connector](https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/connectors/memory/mongodb_atlas).

This tutorial takes you through how to use Microsoft Semantic Kernel to properly upload and embed documents into your MongoDB Atlas cluster, and then conduct queries using Microsoft Semantic Kernel as well, all in Python!

## Pre-requisites 

 - [MongoDB Atlas cluster](https://www.mongodb.com/docs/atlas/getting-started/) 
 - IDE of your choice (this tutorial uses [Google Colab](https://colab.research.google.com/drive/1Qlgor8-nFEkugMdTADYVKIYZejSAWQUV#scrollTo=U8u9vfQYnWvt) — please refer to it if you’d like to run the commands directly)
 - [OpenAI API key](https://platform.openai.com/docs/introduction)

Let’s get started!

## Setting up our Atlas cluster
Visit the [MongoDB Atlas](https://account.mongodb.com/account/login) dashboard and set up your cluster. In order to take advantage of the `$vectorSearch` operator in an aggregation pipeline, you need to run MongoDB Atlas 6.0.11 or higher. This tutorial can be built using a free cluster. 

When you’re setting up your deployment, you’ll be prompted to set up a database user and rules for your network connection. Please ensure you save your username and password somewhere safe and have the correct IP address rules in place so your cluster can connect properly. 

If you need more help getting started, check out our [tutorial on MongoDB Atlas](https://www.mongodb.com/basics/mongodb-atlas-tutorial). 

## Installing the latest version of Semantic Kernel
In order to be successful with our tutorial, let’s ensure we have the most up-to-date version of Semantic Kernel installed in our IDE. As of the creation of this tutorial, the latest version is 0.3.14. Please run this `pip` command in your IDE to get started:
```
!python -m pip install semantic-kernel==0.3.14.dev
```
Once it has been successfully run, you will see various packages being downloaded. Please ensure `pymongo` is downloaded in this list. 

## Setting up our imports 
Here, include the information about our OpenAI API key and our connection string. 

Let’s set up the necessary imports:
```
import openai
import semantic_kernel as sk
from semantic_kernel.connectors.ai.open_ai import OpenAIChatCompletion, OpenAITextEmbedding
from semantic_kernel.connectors.memory.mongodb_atlas import MongoDBAtlasMemoryStore



kernel = sk.Kernel()


openai.api_key = '<your-openai-key'
MONGODB_SEARCH_INDEX="defaultRandomFacts"
MONGODB_DATABASE="semantic-kernel"
MONGODB_COLLECTION="randomFacts"


kernel.add_chat_service("chat-gpt", OpenAIChatCompletion("gpt-3.5-turbo", openai.api_key))
kernel.add_text_embedding_generation_service("ada", OpenAITextEmbedding("text-embedding-ada-002", openai.api_key))


kernel.register_memory_store(memory_store=MongoDBAtlasMemoryStore(index_name=MONGODB_SEARCH_INDEX, database_name=MONGODB_DATABASE, connection_string="<mongodb-uri>"))
kernel.import_skill(sk.core_skills.TextMemorySkill())
```
Importing in OpenAI is crucial because we are using their data model to embed not only our documents but also our queries. We also want to import their Text Embedding library for this same reason. For this tutorial, we are using the embedding model `ada-002`, but please double check that you’re using a model that is compatible with your OpenAI API key.

Our `MongoDBAtlasMemoryStore` class is very important as it’s the part that enables us to use MongoDB as our memory store. This means we can connect to the Semantic Kernel and have our documents properly saved and formatted in our cluster. For more information on this class, please refer to the [repository](https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/connectors/memory/mongodb_atlas). 

This is also where you will need to incorporate your OpenAI API key along with your [MongoDB connection string](https://www.mongodb.com/docs/atlas/), and other important variables that we will use. The ones above are just a suggestion, but if they are changed while attempting the tutorial, please ensure they are consistent throughout. For help on accessing your OpenAI key, please read the section below.

### Generate your OpenAI key
In order to generate our embeddings, we will use the OpenAI API. First, we’ll need a secret key. To create your OpenAI key, you'll need to create an account. Once you have that, visit the [OpenAI API](https://platform.openai.com/) and you should be greeted with a screen like the one below. Click on your profile icon in the top right of the screen to get the dropdown menu and select “View API keys”.
![where to find your API key on OpenAI](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltbb8293d6a59ed869/65568a482e0ea1959117960a/semantickernela1.png)

Here, you can generate your own API key by clicking the “Create new secret key” button. Give it a name and store it somewhere safe. This is all you need from OpenAI to use their API to generate your embeddings.

![how to create a new API key](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1550895c161b4e43/65568a9b4452cc73ecf967a6/semantickernela2.png)

## The need for retrieval-augmented generation (RAG)
Retrieval-augmented regeneration, also known as RAG, is an NLP technique that can help improve the quality of large language models (LLMs). It’s an artificial intelligence framework for getting data from an external knowledge source. The memory store we are creating using Microsoft Semantic Kernel is an example of this. But why is RAG necessary? Let’s take a look at an example.

LLMs like OpenAI GPT-3.5 exhibit an impressive and wide range of skills. They are trained on the data available on the internet about a wide range of topics and can answer queries accurately. Using Semantic Kernel, let’s ask OpenAI’s LLM if Albert Einstein likes coffee: 
```
# Wrap your prompt in a function
prompt = kernel.create_semantic_function("""
As a friendly AI Copilot, answer the question: Did Albert Einstein like coffee?
""")


print(prompt())
```
The output received is:
```
Yes, Albert Einstein was known to enjoy coffee. He was often seen with a cup of coffee in his hand and would frequently visit cafes to discuss scientific ideas with his colleagues over a cup of coffee.
```
Since this information was available on the public internet, the LLM was able to provide the correct answer.

But LLMs have their limitations: They have a knowledge cutoff (September 2021, in the case of OpenAI) and do not know about proprietary and personal data. They also have a tendency to hallucinate — that is, they may confidently make up facts and provide answers that may seem to be accurate but are actually incorrect. Here is an example to demonstrate this knowledge gap:

```
prompt = kernel.create_semantic_function("""
As a friendly AI Copilot, answer the question: Did I like coffee?
""")


print(prompt())
```
The output received is:
```
As an AI, I don't have personal preferences or experiences, so I can't say whether "I" liked coffee or not. However, coffee is a popular beverage enjoyed by many people around the world. It has a distinct taste and aroma that some people find appealing, while others may not enjoy it as much. Ultimately, whether someone likes coffee or not is a subjective matter and varies from person to person.
``` 

As you can see, there is a knowledge gap here because we don’t have our personal data loaded in OpenAI that our query can access. So let’s change that. Continue on through the tutorial to learn how to augment the knowledge base of the LLM with proprietary data. 

## Add some documents into our MongoDB cluster
Once we have incorporated our MongoDB connection string and our OpenAI API key, we are ready to add some documents into our MongoDB cluster. 

Please ensure you’re specifying the proper collection variable below that we set up above. 
```
async def populate_memory(kernel: sk.Kernel) -> None:
# Add some documents to the semantic memory
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="1", text="We enjoy coffee and Starbucks"
)
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="2", text="We are Associate Developer Advocates at MongoDB"
)
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="3", text="We have great coworkers and we love our teams!"
)
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="4", text="Our names are Anaiya and Tim"
)
await kernel.memory.save_information_async(
collection=MONGODB_COLLECTION, id="5", text="We have been to New York City and Dublin"
)
```
Here, we are using the `populate_memory` function to define five documents with various facts about Anaiya and Tim. As you can see, the name of our collection is called “randomFacts”, we have specified the ID for each document (please ensure each ID is unique, otherwise you will get an error), and then we have included a text phrase we want to embed. 

Once you have successfully filled in your information and have run this command, let’s add them to our cluster — aka let’s populate our memory! To do this, please run the command:
```
print("Populating memory...aka adding in documents")
await populate_memory(kernel)
```
Once this command has been successfully run, you should see the database, collection, documents, and their embeddings populate in your Atlas cluster. The screenshot below shows how the first document looks after running these commands.

![screenshot of our first document with the text embedding](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0a505f52e4d85526/65568b25a255e23bc9211f7a/semantickernela3.png)
  
Once the documents added to our memory have their embeddings, let’s set up our search index and ensure we can generate embeddings for our queries. 

## Create a vector search index in MongoDB 
In order to use the `$vectorSearch` operator on our data, we need to set up an appropriate search index. We’ll do this in the [Atlas UI](https://account.mongodb.com/account/login). Select the “Search" tab on your cluster and click “Create Search Index”.
![where to create a vector search index](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4fb2a898919fdbc2/65568b844e0195c4c3bf1f64/semantickernela4.png)

We want to choose the "JSON Editor Option" and click "Next".

![specifics for creating a search index](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt33055bae582002e4/655bb51b3133dbdcc8911770/semantickernel5.png)

On this page, we're going to select our target database, `semantic-kernel`, and collection, `randomFacts`.

For this tutorial, we are naming our index `defaultRandomFacts`. The index will look like this: 

```json
{
  "mappings": {
    "dynamic": true,
    "fields": {
      "embedding": {
        "dimensions": 1536,
        "similarity": "dotProduct",
        "type": "knnVector"
      }
    }
  }
}
```
The fields specify the embedding field name in our documents, `embedding`, the dimensions of the model used to embed, `1536`, and the similarity function to use to find K-nearest neighbors, `dotProduct`. It's very important that the dimensions in the index match that of the model used for embedding. This data has been embedded using the same model as the one we'll be using, but other models are available and may use different dimensions.

Check out our [Vector Search documentation](https://www.mongodb.com/docs/atlas/atlas-search/vector-search/) for more information on the index configuration settings.

## Query documents using Microsoft Semantic Kernel
In order to query your new documents hosted in your MongoDB cluster “memory” store, we can use the `memory.search_async` function. Run the following commands and watch the magic happen:

```
result = await kernel.memory.search_async(MONGODB_COLLECTION, 'What is my job title?')


print(f"Retrieved document: {result[0].text}, {result[0].relevance}")
```

Now you can ask any question and get an accurate response!  

Examples of questions asked and the results: 
![the result of the question: What is my job title?](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb0ffef5ac86f7f5c/655bb5826f7bf46d9a82c7fe/semantickernel6.png)

![the result of the question: Do I like caffeine?](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4dcf417349dfc947/655bb5b4867c0b640138726c/semantickernel7.png)

![the result of the question: Where in the world have I visited?](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3f87d869993ff195/655bb5e17c56dde9cba5dcac/semantickernel8.png)

## Conclusion
In this tutorial, you have learned a lot of very useful concepts:

 - What Microsoft Semantic Kernel is and why it’s important.
 - How to connect Microsoft Semantic Kernel to a MongoDB Atlas cluster.
 - How to add in documents to your MongoDB memory store (and embed them, in the process, through Microsoft Semantic Kernel).
 - How to query your new documents in your memory store using Microsoft Semantic Kernel.

For more information on MongoDB Vector Search, please visit the [documentation](https://www.mongodb.com/products/platform/atlas-vector-search), and for more information on Microsoft Semantic Kernel, please visit their [repository](https://github.com/microsoft/semantic-kernel) and [resources](https://learn.microsoft.com/en-us/semantic-kernel/overview/). 

If you have any questions, please visit our [MongoDB Developer Community Forum](https://www.mongodb.com/community/forums/). 

Follow this comprehensive guide to getting started with Microsoft Semantic Kernel and MongoDB Atlas Vector Search.

Building AI Applications with Microsoft Semantic Kernel and MongoDB Atlas Vector Search

Ken W. Alger

As an avid traveler, you have a love for staying at Airbnbs and have been keeping detailed notes about each one you’ve stayed in over the years. These notes are spread out across different storage locations, like MongoDB Atlas and AWS S3, making it a challenge to search for a specific Airbnb with the amenities your girlfriend desires for your upcoming Valentine’s Day trip. Luckily, there is a solution to make this process a lot easier. By using MongoDB’s Data Federation feature, you can combine all your data into one logical view and easily search for the perfect Airbnb without having to worry about where the data is stored. This way, you can make your Valentine’s Day trip perfect without wasting time searching through different databases and storage locations.

Don’t know how to utilize MongoDB’s Data Federation feature? This tutorial will guide you through exactly how to combine your Airbnb data together for easier query-ability.

## Tutorial Necessities

Before we jump in, there are a few necessities we need to have in order to be on the same page. This tutorial requires:

* MongoDB Atlas.
* An Amazon Web Services (AWS) account.
* Access to the AWS Management Console.
* AWS CLI.
* MongoDB Compass.

### Importing our sample data

Our first step is to import our Airbnb data into our Atlas cluster and our S3 bucket, so we have data to work with throughout this tutorial. Make sure to import the dataset into both of these storage locations.

### Importing via MongoDB Atlas

Step 1: Create a [free tier shared cluster](https://www.mongodb.com/basics/clusters/mongodb-cluster-setup).

Step 2: Once your cluster is set up, click the three ellipses and click “Load Sample Dataset."
![loading sample dataset](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/loading_sample_dataset_7433def6ca.png)

Step 3: Once you get this green message you’ll know your sample dataset (Airbnb notes) is properly loaded into your cluster.
![sample dataset successfull loaded](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/sample_dataset_successfully_loaded_3ad17e57bb.png)

### Importing via AWS S3
Step 1: We will be using [this sample data set](https://developer-advocacy-public.s3-eu-west-1.amazonaws.com/blog/sampledata.zip). Please download it locally. It contains the sample data we are working with along with the S3 bucket structure necessary for this demo. 

Step 2: Once the data set is downloaded, access your AWS Management Console and navigate to their S3 service. 

Step 3: Hit the button “Create Bucket” and [follow the instructions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to create your bucket and upload the sampledata.zip. 

Step 4: Make sure to unzip your file before uploading the folders into S3. 

Step 5: Once your data is loaded into the bucket, you will see several folders, each with varying data types.

![folders with data types](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/folders_with_data_types_3c53eeeefa.png)

Step 6: Follow the path: Amazon S3 > Buckets > atlas-data-federation-demo > json/ > airbnb/ to view your Airbnb notes. Your bucket structure should look like this:

![bucket structure](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/bucket_structure_4852f173bc.png)

Congratulations! You have successfully uploaded your extensive Airbnb notes in not one but two storage locations. Now, let’s see how to retrieve this information in one location using Data Federation so we can find the perfect Airbnb. In order to do so, we need to get comfortable with the MongoDB Atlas Data Federation console. 

## Connecting MongoDB Atlas to S3

Inside the MongoDB Atlas console, on the left side, click on Data Federation.

![Data Federation](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc94edfdcc342265b/649c786b84a4c7eb46d8db62/data-federation.png)

Here, click “set up manually” in the "create new federated database" dropdown in the top right corner of the UI. This will lead us to a page where we can add in our data sources. You can rename your Federated Database Instance to anything you like. Once you save it, you will not be able to change the name.

![Naming your Data Federation instance](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/naming_data_federation_instance_65ee5be371.png)

Let’s add in our data sources from our cluster and our bucket!

### Adding in data source via AWS S3 Bucket:
Step 1: Click on “Add Data Source.” 

Step 2: Select the “Amazon S3” button and hit “Next.” 

![Adding a data source](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/adding_data_source_ef7ae8478d.png)

Step 3: From here, click Next on the “Authorize an AWS IAM Role”:

![Configure AWS S3 data store](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/configure_data_store_0c31660f41.png)

Step 4: Click on “Create New Role in the AWS CLI”: 

![Creating a new role](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/creating_a_new_role_4a772bcd89.png)

Step 5: Now, you’re going to want to make sure you have AWS CLI configured on your laptop. 

Step 6: Follow the steps below the “Create New Role with the AWS CLI” in your AWS CLI. 

```
aws iam create-role \
 --role-name datafederation \
 --assume-role-policy-document file://role-trust-policy.json
```

Step 7: You can find your “ARN” directly in your terminal. Copy that in — it should look like this:

```
arn:aws:iam::7***************:role/datafederation
```

Step 8: Enter the bucket name containing your Airbnb notes:

![Enter bucket name containing Airbnb notes](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/enter_bucket_name_2b78f110b3.png)

Step 9: Follow the instructions in Atlas and save your policy role. 

![Save policy role](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/save_policy_role_4315553ee6.png)

Step 10: Copy the CLI commands listed on the screen and paste them into your terminal like so:

```
aws iam put-role-policy \
  --role-name datafederation \
  --policy-name datafederation-policy \
  --policy-document file://adl-s3-policy.json
```

Step 11: Access your AWS Console, locate your listingsAndReviews.json file located in your S3 bucket, and copy the S3 URI. 

![Copy S3 URI](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/copy_s3_uri_e12ada2d17.png)

Step 12: Enter it back into your “Define ‘Data Sources’ Using Paths Inside Your S3” screen and change each step of the tree to “static.” 

![Changing steps to static](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/changing_steps_to_static_75b0b6ab43.png)

Step 13:  Drag your file from the left side of the screen to the middle where it says, “Drag the dataset to your Federated Database.” Following these steps correctly will result in a page similar to the screenshot below.

![Data source to virtual connection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/data_source_to_virtual_connection_6e7e3a7252.png)

You have successfully added in your Airbnb notes from your S3 bucket. Nice job. Let's do the same thing for the notes saved in our Atlas cluster. 

### Adding in data source via MongoDB Atlas cluster
Step 1: Click “Add Data Sources.”

Step 2: Select “MongoDB Atlas Cluster” and provide the cluster name along with our sample_airbnb collection. These are your Atlas Airbnb notes. 

![Provide cluster name](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/provide_cluster_name_f5cd029131.png)

Step 3: Click “Next” and your sample_airbnb.listingsAndReviews will appear in the left-hand side of the console.

Step 4: Drag it directly under your Airbnb notes from your S3 bucket and hit “Save.” Your console should look like this when done:

![Saving S3 bucket](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/dragging_file_40f01a6f88.png)

Great job. You have successfully imported your Airbnb notes from both your S3 bucket and your Atlas cluster into one location. Let’s connect to our Federated Database and see our data combined in one easily query-able location. 

## Connect to your federated database
We are going to connect to our Federated Database using MongoDB Compass. 

Step 1: Click the green “Connect” button and then select “Connect using MongoDB Compass.” 

![Connect to FederationDemo](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/connect_to_federation_demo_769733e967.png)

Step 2: Copy in the connection string, making sure to switch out the user and password for your own. This user must have admin access in order to access the data. 

Step 3: Once you’re connected to Compass, click on “VirtualDatabase0” and once more on “VirtualCollection0.” 

![virtual database](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/virtual_database_c41050af02.png)

Amazing job. You can now look at all your Airbnb notes in one location! 

## Conclusion

In this tutorial, we have successfully stored your Airbnb data in various storage locations, combined these separate data sets into one via Data Federation, and successfully accessed our data back through MongoDB Compass. Now you can look for and book the perfect Airbnb for your trip in a fraction of the time. 

This tutorial will guide you through exactly how to combine your Airbnb data together for easier query-ability. 

MongoDB Data Federation Setup

Atlas Vector Search was recently released, so let’s dive into a tutorial on how to properly model your documents when utilizing vector search to revolutionize your querying capabilities!

## Data modeling normally in MongoDB

[Vector search](https://www.mongodb.com/products/platform/atlas-vector-search) is new, so let’s first go over the basic ways of modeling your data in a MongoDB document before continuing on into how to incorporate vector embeddings. 

Data modeling in MongoDB revolves around organizing your data into documents within various collections. Varied projects or organizations will require different ways of structuring data models due to the fact that successful data modeling depends on the specific requirements of each application, and for the most part, no one document design can be applied for every situation. There are some commonalities, though, that can guide the user. These are:

 1. Choosing whether to embed or reference your related data. 
 2. Using arrays in a document.
 3. Indexing your documents (finding fields that are frequently used and applying the appropriate indexing, etc.).

For a more in-depth explanation and a comprehensive guide of data modeling with MongoDB, please check out our [data modeling article](https://www.mongodb.com/basics/data-modeling).

## Setting up an example data model

We are going to be building our vector embedding example using a MongoDB document for our [MongoDB TV](https://www.youtube.com/user/mongodb) series. Here, we have a single MongoDB document representing our MongoDB TV show, without any embeddings in place. We have a nested array featuring our array of seasons, and within that, our array of different episodes. This way, in our document, we are capable of seeing exactly which season each episode is a part of, along with the episode number, the title, the description, and the date: 

```
{
   "_id": ObjectId("238478293"),
   "title": "MongoDB TV",
   "description": "All your MongoDB updates, news, videos, and podcast episodes, straight to you!",
   "genre": ["Programming", "Database", "MongoDB"],
   "seasons": [
      {
         "seasonNumber": 1,
         "episodes": [
            {
               "episodeNumber": 1,
               "title": "EASY: Build Generative AI Applications",
               "description": "Join Jesse Hall….",
               "date": ISODate("Oct52023")
            },
            {
               "episodeNumber": 2,
               "title": "RAG Architecture & MongoDB: The Future of Generative AI Apps",
               "description": "Join Prakul Agarwal…",
               "date": ISODate("Oct42023")
            }
         ]
      },
      {
         "seasonNumber": 2,
         "episodes": [
            {
               "episodeNumber": 1,
               "title": "Cloud Connect - Harness the Power of AI/ML and Generative AI on AWS with MongoDB Atlas",
               "description": "Join Igor Alekseev….",
               "date": ISODate("Oct32023")
            },
            {
               "episodeNumber": 2,
               "title": "The Index: Here’s what you missed last week…",
               "description": "Join Megan Grant…",
               "date": ISODate("Oct22023")
            }
         ]
      }
   ]
}
```

Now that we have our example set up, let’s incorporate vector embeddings and discuss the proper techniques to set you up for success.

## Integrating vector embeddings for vector search in our data model 

Let’s first understand exactly what vector search is: Vector search is the way to search based on *meaning* rather than specific words. This comes in handy when querying using similarities rather than searching based on keywords. When using vector search, you can query using a question or a phrase rather than just a word. In a nutshell, vector search is great for when you can’t think of *exactly* that book or movie, but you remember the plot or the climax. 

This process happens when text, video, or audio is transformed via an encoder into vectors. With MongoDB, we can do this using OpenAI, Hugging Face, or other natural language processing models. Once we have our vectors, we can upload them in the base of our document and conduct vector search using them. Please keep in mind the [current limitations](https://www.mongodb.com/docs/atlas/atlas-search/field-types/knn-vector/) of vector search and how to properly embed your vectors. 

You can store your vector embeddings alongside other data in your document, or you can store them in a new collection. It is really up to the user and the project goals. Let’s go over what a document with vector embeddings can look like when you incorporate them into your data model, using the same example from above: 

```
{
   "_id": ObjectId("238478293"),
   "title": "MongoDB TV",
   "description": "All your MongoDB updates, news, videos, and podcast episodes, straight to you!",
   "genre": ["Programming", "Database", "MongoDB"],
   “vectorEmbeddings”: [ 0.25, 0.5, 0.75, 0.1, 0.1, 0.8, 0.2, 0.6, 0.6, 0.4, 0.9, 0.3, 0.2, 0.7, 0.5, 0.8, 0.1, 0.8, 0.2, 0.6 ],
   "seasons": [
      {
         "seasonNumber": 1,
         "episodes": [
            {
               "episodeNumber": 1,
               "title": "EASY: Build Generative AI Applications",
               "description": "Join Jesse Hall….",
               "date": ISODate("Oct 5, 2023")
 	
            },
            {
               "episodeNumber": 2,
               "title": "RAG Architecture & MongoDB: The Future of Generative AI Apps",
               "description": "Join Prakul Agarwal…",
               "date": ISODate("Oct 4, 2023")
            }
         ]
      },
      {
         "seasonNumber": 2,
         "episodes": [
            {
               "episodeNumber": 1,
               "title": "Cloud Connect - Harness the Power of AI/ML and Generative AI on AWS with MongoDB Atlas",
               "description": "Join Igor Alekseev….",
               "date": ISODate("Oct 3, 2023")
            },
            {
               "episodeNumber": 2,
               "title": "The Index: Here’s what you missed last week…",
               "description": "Join Megan Grant…",
               "date": ISODate("Oct 2, 2023")
            }
         ]
      }
   ]
}
```
Here, you have your vector embeddings classified at the base in your document. Currently, there is a limitation where vector embeddings cannot be nested in an array in your document. Please ensure your document has your embeddings at the base. There are various tutorials on our [Developer Center](https://www.mongodb.com/developer/products/atlas/building-generative-ai-applications-vector-search-open-source-models/), alongside our [YouTube account](https://www.youtube.com/watch?v=1ZIYVNvRVsY) and our [documentation](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/), that can help you figure out how to embed these vectors into your document and how to acquire the necessary vectors in the first place. 

## Extras: Indexing with vector search

When you’re using vector search, it is necessary to create a search index so you’re able to be successful with your semantic search. To do this, please view our [Vector Search documentation](https://www.mongodb.com/docs/atlas/atlas-search/field-types/knn-vector/). Here is the skeleton code provided by our documentation:

```
{
  "mappings": {
    "dynamic": true|false,
    "fields": {
      "<field-name>": {
        "type": "knnVector",
        "dimensions": <number-of-dimensions>,
        "similarity": "euclidean | cosine | dotProduct"
      }
    }
  }
}
```

When setting up your search index, you want to make sure to click either “true” or “false” for dynamic. (Normally, it’s true.) You want to change the “<field-name>” to be your vector path. In our case, it would be “vectorEmbeddings”. “type” can stay the way it is. For “dimensions”, please match the dimensions of the model you’ve chosen. This is just the number of vector dimensions, and the value cannot be greater than 2048. This limitation comes from the base embedding model that is being used, so please ensure you’re using a supported LLM (large language model) such as OpenAI or Hugging Face. When using one of these, there won’t be any issues running into vector dimensions. For “similarity”, please pick which vector function you want to use to search for the top K-nearest neighbors. 

## Extras: Querying with vector search

When you’re ready to query and find results from your embedded documents, it’s time to create an aggregation pipeline on your embedded vector data. To do this, you can use the“$vectorSearch” operator, which is a new aggregation stage in Atlas. It helps execute an Approximate Nearest Neighbor query. 

For more information on this step, please check out the tutorial on Developer Center about [building generative AI applications](https://www.mongodb.com/developer/products/atlas/building-generative-ai-applications-vector-search-open-source-models/), and our [YouTube video](https://www.youtube.com/watch?v=1ZIYVNvRVsY) on vector search.



Follow along with this comprehensive tutorial on how to properly model your documents for MongoDB Vector Search.

How to Model Your Documents for Vector Search

Data processing is no easy feat, but with the proper tools, it can be simplified and can enable you to make the best data-driven decisions possible. In a world overflowing with data, we need the best methods to derive the most useful information. 

The combination of [MongoDB Atlas](https://www.mongodb.com/atlas/database) with [Azure Databricks](https://learn.microsoft.com/en-us/azure/databricks/introduction/) makes an efficient choice for big data processing. By connecting Atlas with Azure Databricks, we can extract data from our Atlas cluster, process and analyze the data using [PySpark](https://spark.apache.org/docs/latest/api/python/), and then store the processed data back in our Atlas cluster. Using Azure Databricks to analyze your Atlas data allows for access to Databricks’ wide range of advanced analytics capabilities, which include machine learning, data science, and areas of artificial intelligence like natural language processing! Processing your Atlas data with these advanced Databricks tools allows us to be able to handle any amount of data in an efficient and scalable way, making it easier than ever to gain insights into our data sets and enable us to make the most effective data-driven decisions. 

This tutorial will show you how to utilize PySpark to connect Atlas with Databricks so you can take advantage of both platforms. 

MongoDB Atlas is a scalable and flexible storage solution for your data while Azure Databricks provides the power of Apache Spark to work with the security and collaboration features that are available with a Microsoft Azure subscription. [Apache Spark](https://spark.apache.org/) provides the Python interface for working with Spark, PySpark, which allows for an easy-to-use interface for developing in Python. To properly connect PySpark with MongoDB Atlas, the [MongoDB Spark Connector](https://www.mongodb.com/docs/spark-connector/current/) is utilized. This connector ensures for seamless compatibility, as you will see below in the tutorial. 

![Alt Text: technical diagram showing how Atlas, PySpark, the MongoDB Spark Connector, and Azure Databricks are integrated](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/databrickspic_6abce4187c.png)

Our tutorial to combine the above platforms will consist of viewing and manipulating an Atlas cluster and visualizing our data from the cluster back in our PySpark console. We will be setting up both Atlas and Azure Databricks clusters, connecting our Databricks cluster to our IDE, and writing scripts to view and contribute to the cluster in our Atlas account. Let’s get started!

### Requirements
In order to successfully recreate this project, please ensure you have everything in the following list: 

* [MongoDB Atlas](https://www.mongodb.com/atlas/database) account.

* [Microsoft Azure](https://azure.microsoft.com/en-us/products/databricks/) subscription (two-week free tier trial). 

* Python 3.8+.

* [GitHub Repository](https://github.com/mongodb-developer/atlas-pyspark-databricks).

* Java on your local machine.

## Setting up a MongoDB Atlas cluster
Our first step is to set up a MongoDB Atlas cluster. Access the Atlas UI and [follow these steps](https://www.mongodb.com/basics/clusters/mongodb-cluster-setup). For this tutorial, a free “shared” cluster is perfect. Create a database and name it “bookshelf” with a collection inside named “books”. To ensure ease for this tutorial, please allow for a connection from anywhere within your cluster’s network securities.

Once properly provisioned, your cluster will look like this:
![Alt Text: the initial set-up of our Atlas cluster](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/databricks1_f1bb39aaec.png)

Now we can set up our Azure Databricks cluster. 

## Setting up an Azure Databricks cluster

Access the [Azure Databricks](https://azure.microsoft.com/en-us/products/databricks/) page, sign in, and access the Azure Databricks tab. This is where you’ll create an Azure Databricks workspace. 

![Alt Text: the configuration of our Azure Databricks workspace](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2023_05_01_at_2_58_42_PM_f44cb7b2ab.png)

For our Databricks cluster, a free trial works perfectly for this tutorial. Once the cluster is provisioned, you’ll only have two weeks to access it before you need to upgrade. 

Hit “Review and Create” at the bottom. Once your workspace is validated, click “Create.” Once your deployment is complete, click on “Go to Resource.” You’ll be taken to your workspace overview. Click on “Launch Workspace” in the middle of the page. 

This will direct you to the Microsoft Azure Databricks UI where we can create the Databricks cluster. On the left-hand of the screen, click on “Create a Cluster,” and then click “Create Compute” to access the correct form. 

![Alt Text: the configuration of our Azure Databricks cluster](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2023_05_01_at_3_00_13_PM_dcf3d34ade.png)

When creating your cluster, pay close attention to what your “Databricks runtime version” is. Continue through the steps to create your cluster. 

We’re now going to install the libraries we need in order to connect to our MongoDB Atlas cluster. Head to the “Libraries” tab of your cluster, click on “Install New,” and select “Maven.” Hit “Search Packages” next to “Coordinates.” Search for `mongo` and select the `mongo-spark` package. Do the same thing with `xml` and select the `spark-xml` package. When done, your library tab will look like this: 
![Alt Text: libraries to install for your Databricks cluster](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2023_05_01_at_3_02_13_PM_5439367d65.png)

## Utilizing Databricks-Connect
Now that we have our Azure Databricks cluster ready, we need to properly connect it to our IDE. We can do this through a very handy configuration named Databricks Connect. Databricks Connect allows for Azure Databricks clusters to connect seamlessly to the IDE of your choosing.

### Databricks configuration essentials

Before we establish our connection, let’s make sure we have our configuration essentials. This is available in [the Databricks Connect tutorial](https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect) on Microsoft’s website under “Step 2: Configure connection properties.” Please note these properties down in a safe place, as you will not be able to connect properly without them.


### Databricks-Connect configuration
Access the Databricks Connect page linked above to properly set up `databricks-connect` on your machine. Ensure that you are downloading the `databricks-connect` version that is compatible with your Python version and is the same as the Databricks runtime version in your Azure cluster. 

>Please ensure prior to installation that you are working with a virtual environment for this project. Failure to use a virtual environment may cause PySpark package conflicts in your console. 

Virtual environment steps in Python:
```
python3 -m venv name 
```
Where the `name` is the name of your environment, so truly you can call it anything. 

Our second step is to activate our virtual environment:
```
source name/bin/activate 
```
And that’s it. We are now in our Python virtual environment. You can see that you’re in it when the little (name) or whatever you named it shows up.

* * *

Continuing on...for our project, use this installation command:
```
pip install -U “databricks-connect==10.4.*” 
```
Once fully downloaded, we need to set up our cluster configuration. Use the configure command and follow the instructions. This is where you will input your configuration essentials from our “Databricks configuration essentials” section.

Once finished, use this command to check if you’re connected to your cluster:
```
databricks-connect test
```

You’ll know you’re correctly configured when you see an “All tests passed” in your console. 
Now, it’s time to set up our SparkSessions and connect them to our Atlas cluster.

## SparkSession + Atlas configuration
The creation of a SparkSession object is crucial for our tutorial because it provides a way to access all important PySpark features in one place. These features include: reading data, creating data frames, and managing the overall configuration of PySpark applications. Our SparkSession will enable us to read and write to our Atlas cluster through the data frames we create.

The full code is on our [Github](https://github.com/mongodb-developer/atlas-pyspark-databricks) account, so please access it there if you would like to replicate this exact tutorial. We will only go over the code for some of the essentials of the tutorial below.

This is the SparkSession object we need to include. We are going to use a basic structure where we describe the application name, configure our “read” and “write” connectors to our `connection_string` (our MongoDB cluster [connection string](https://www.mongodb.com/docs/guides/atlas/connection-string/) that we have saved safely as an environment variable), and configure our `mongo-spark-connector`. Make sure to use the correct `mongo-spark-connector` for your environment. For ours, it is version 10.0.3. Depending on your Python version, the `mongo-spark-connector` version might be different. To find which version is compatible with your environment, please refer to the [MVN Repository documents](https://mvnrepository.com/artifact/org.mongodb.spark/mongo-spark-connector). 

```
# use environment variable for uri 
load_dotenv()
connection_string: str = os.environ.get("CONNECTION_STRING")

# Create a SparkSession. Ensure you have the mongo-spark-connector included.
my_spark = SparkSession \
    .builder \
    .appName("tutorial") \
    .config("spark.mongodb.read.connection.uri", connection_string) \
    .config("spark.mongodb.write.connection.uri", connection_string) \
    .config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector:10.0.3") \
    .getOrCreate()

```

For more help on how to create a SparkSession object with MongoDB and for more details on the `mongo-spark-connector`, please view the [documentation](https://www.mongodb.com/docs/spark-connector/current/getting-started/).

Our next step is to create two data frames, one to `write` a book to our Atlas cluster, and a second to `read` back all the books in our cluster. These data frames are essential; make sure to use the proper format or else they will not properly connect to your cluster. 

[Data frame to `write`](https://www.mongodb.com/docs/spark-connector/current/write-to-mongodb/) a book:
```
add_books = my_spark \
    .createDataFrame([("<title>", "<author>", <year>)], ["title", "author", "year"])

add_books.write \
    .format("com.mongodb.spark.sql.DefaultSource") \
    .option('uri', connection_string) \
    .option('database', 'bookshelf') \
    .option('collection', 'books') \
    .mode("append") \
    .save() 

```
[Data frame to `read`](https://www.mongodb.com/docs/spark-connector/current/read-from-mongodb/) back our books:
```
# Create a data frame so you can read in your books from your bookshelf.
return_books = my_spark.read.format("com.mongodb.spark.sql.DefaultSource") \
    .option('uri', connection_string) \
    .option('database', 'bookshelf') \
    .option('collection', 'books') \
    .load()

# Show the books in your PySpark shell.
return_books.show()

```

Add in the book of your choosing under the `add_books` dataframe. Here, exchange the title, author, and year for the areas with the `< >` brackets. Once you add in your book and run the file, you’ll see that the logs are telling us we’re connecting properly and we can see the added books in our PySpark shell. This demo script was run six separate times to add in six different books. A picture of the console is below:

![Alt Text: list of books from our Atlas cluster shown in our PySpark shell](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/databricksbooksconsole_3b5f6102fc.png)

We can double-check our cluster in Atlas to ensure they match up: 
![Alt Text: list of books in our Atlas cluster shown in our Atlas account](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2023_05_01_at_2_46_38_PM_98f6cfe75b.png)

## Conclusion
Congratulations! We have successfully connected our MongoDB Atlas cluster to Azure Databricks through PySpark, and we can `read` and `write` data straight to our Atlas cluster. 

The skills you’ve learned from this tutorial will allow you to utilize Atlas’s scalable and flexible storage solution while leveraging Azure Databricks’ advanced analytics capabilities. This combination can allow developers to handle any amount of data in an efficient and scalable manner, while allowing them to gain insights into complex data sets to make exciting data-driven decisions! 

Questions? Comments? Let’s continue the conversation over at the [MongoDB Developer Community](https://www.mongodb.com/community/forums/)!

This tutorial will show you how to connect MongoDB Atlas to Azure Databricks using PySpark. 


 Utilizing PySpark to Connect MongoDB Atlas with Azure Databricks

Picture a developer’s paradise: a world where instead of fussing over hardware complexities, we are free to focus entirely on running and executing our applications. With the combination of [AWS Lambda](https://aws.amazon.com/lambda/) and [MongoDB Atlas](https://www.mongodb.com/products/platform/cloud#atlasMDB?utm_campaign=devrel&utm_source=cross-post&utm_term=anaiya.raisinghani), this vision becomes a reality. 

Armed with AWS Lambda’s pay-per-execution structure and MongoDB Atlas’ unparalleled scalability, developers will truly understand what it means for their applications to thrive without the hardware limitations they might be used to. 

This tutorial will take you through how to properly set up an Atlas cluster, connect it to AWS Lambda using [MongoDB’s Python Driver](https://www.mongodb.com/docs/drivers/pymongo/?utm_campaign=devrel&utm_term=anaiya.raisinghani), write an aggregation pipeline on our data, and return our wanted information. Let’s get started. 

### Prerequisites for success
* [MongoDB Atlas Account](https://www.mongodb.com/cloud/atlas/register?utm_campaign=devrel&utm_source=cross-post&utm_term=anaiya.raisinghani)
* [AWS Account](https://aws.amazon.com/account/); Lambda access is necessary
* GitHub [repository](https://github.com/mongodb-developer/awslambda-pymongo)
* Python 3.8+

## Create an Atlas Cluster
Our first step is to create an Atlas cluster. Log into the Atlas UI and follow the [steps](https://www.mongodb.com/basics/clusters/mongodb-cluster-setup?utm_campaign=devrel&utm_source=cross-post&utm_term=anaiya.raisinghani) to set it up. For this tutorial, the free tier is recommended, but any tier will work! 

Please ensure that the cloud provider picked is AWS. It’s also necessary to pick a secure username and password so that we will have the proper authorization later on in this tutorial, along with proper IP address access. 

Once your cluster is up and running, click the ellipses next to the Browse Collections button and download the `sample dataset`. Your finished cluster will look like this:
![fully provisioned Atlas cluster with sample data downloaded](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt6f9a801b00041f85/64a703af1f8d5a624e58764b/Screenshot_2023-07-06_at_2.10.25_PM.png)

Once our cluster is provisioned, let’s set up our AWS Lambda function. 

## Creating an AWS Lambda function
Sign into your AWS account and search for “Lambda” in the search bar. Hit the orange “Create function” button at the top right side of the screen, and you’ll be taken to the image below. Here, make sure to first select the “Author from scratch” option. Then, we want to select a name for our function (AWSLambdaDemo), the runtime (3.8), and our architecture (x86_64). 
![creating our Lambda function](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5f19a8158061b23b/64a705858f332a953d54482d/Screenshot_2023-07-06_at_2.17.34_PM.png)

Hit the orange “Create function” button on the bottom right to continue. Once your function is created, you’ll see a page with your function overview above and your code source right below. 
![AWS Lambda function set up](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt2721b8c101d8d4fd/64a705f74d81430fe78ff398/Screenshot_2023-07-06_at_2.20.28_PM.png)

Now, we are ready to set up our connection from AWS Lambda to our MongoDB cluster.

To make things easier for ourselves because we are going to be using Pymongo, a dependency, instead of editing directly in the code source, we will be using Visual Studio Code. AWS Lambda has a limited amount of pre-installed libraries and dependencies, so in order to get around this and incorporate Pymongo, we will need to package our code in a special way. Due to this “workaround,” this will not be a typical tutorial with testing at every step. We will first have to download our dependencies and upload our code to Lambda prior to ensuring our code works instead of using a typical `requirements.txt` file. More on that below. 

## AWS Lambda and MongoDB cluster connection

Now we are ready to establish a connection between AWS Lambda and our MongoDB cluster! 

Create a new directory on your local machine and name it
    `awslambda-demo`.

 Let’s install `pymongo`. As said above, Lambda doesn’t have every library available. So, we need to download `pymongo` at the root of our project. We can do it by [working with .zip file archives](https://docs.aws.amazon.com/lambda/latest/dg/python-package.html):
In the terminal, enter our `awslambda-demo` directory:
    
     cd awslambda-demo

Create a new directory where your dependencies will live:

    mkdir dependencies

Install `pymongo` directly in your `dependencies` package:

    pip install --target ./dependencies pymongo

Open Visual Studio Code, open the `awslambda-demo` directory, and create a new Python file named `lambda_function.py`. This is where the heart of our connection will be. 

Insert the code below in our `lambda_function.py`. Here, we are setting up our console to check that we are able to connect to our Atlas cluster. Please keep in mind that since we are incorporating our environment variables in a later step, you will not be able to connect just yet. We have copied the `lambda_handler` definition from our Lambda code source and have edited it to insert one document stating my full name into a new “test” database and “test” collection. It is best practice to incorporate our MongoClient outside of our `lambda_handler` because to establish a connection and performing authentication is reactively expensive, and Lambda will re-use this instance.

```
import os
from pymongo import MongoClient


client = MongoClient(host=os.environ.get("ATLAS_URI"))


def lambda_handler(event, context):
    # Name of database
    db = client.test 

    # Name of collection
    collection = db.test 
    
    # Document to add inside
    document = {"first name": "Anaiya", "last name": "Raisinghani"}


    # Insert document
    result = collection.insert_one(document)


    if result.inserted_id:
        return "Document inserted successfully"
    else:
        return "Failed to insert document"
```
If this is properly inserted in AWS Lambda, we will see “Document inserted successfully” and in MongoDB Atlas, we will see the creation of our “test” database and collection along with the single document holding the name “Anaiya Raisinghani.” Please keep in mind we will not be seeing this yet since we haven’t configured our environment variables and will be doing this a couple steps down. 

Now, we need to create a .zip file, so we can upload it in our Lambda function and execute our code. Create a .zip file at the root:

    cd dependencies
    zip -r ../deployment.zip *
This creates a `deployment.zip` file in your project directory.

Now, we need to add in our `lambda_function.py` file to the root of our .zip file:

    cd ..
    zip deployment.zip lambda_function.py
![how our directory is set up](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blte3b31b42ab877c4e/64a70bc682c5c661f5c38578/Screenshot_2023-07-06_at_2.45.12_PM.png)

Once you have your .zip file, access your AWS Lambda function screen, click the “Upload from” button, and select “.zip file” on the right hand side of the page:
![how to upload a .zip file in AWS Lambda](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4343f378c5402722/64a70bf926afdb0077c6fc19/Screenshot_2023-07-06_at_2.46.06_PM.png)

Upload your .zip file and you should see the code from your `lambda_function.py` in your “Code Source”:
![AWS Lambda code source once view](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt465ca3b17b00781d/64a70c3e26afdb1284c6fc1d/Screenshot_2023-07-06_at_2.47.16_PM.png)

Let’s configure our environment variables. Select the “Configuration” tab and then select the “Environment Variables” tab. Here, put in your “ATLAS_URI” string. To access your connection string, please follow the [instructions in our docs](https://www.mongodb.com/docs/guides/atlas/connection-string/?utm_campaign=devrel&utm_source=cross-post&utm_term=anaiya.raisinghani).

![environment variables configuration](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb08cd9dfa2062bd0/64a70c8338fe4382296abbac/Screenshot_2023-07-06_at_2.48.23_PM.png)

Once you have your Environment Variables in place, we are ready to run our code and see if our connection works. Hit the “Test” button. If it’s the first time you’re hitting it, you’ll need to name your event. Keep everything else on the default settings. You should see this page with our “Execution results.” Our document has been inserted!
![After running our code, we can see that it works](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3114cad2f928facb/64a70cd905df43144f5aeb26/Screenshot_2023-07-06_at_2.49.52_PM.png)

When we double-check in Atlas, we can see that our new database “test” and collection “test” have been created, along with our document with “Anaiya Raisinghani.”
![result of our connection shown in MongoDB Atlas](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt2bad4a2e4128a600/64a70d1f23241010123ff625/Screenshot_2023-07-06_at_2.50.57_PM.png)

This means our connection works and we are capable of inserting documents from AWS Lambda to our MongoDB cluster. Now, we can take things a step further and input a simple [aggregation pipeline](https://www.mongodb.com/docs/manual/core/aggregation-pipeline/?utm_campaign=devrel&utm_source=cross-post&utm_term=anaiya.raisinghani)!

## Aggregation pipeline example

For our pipeline, let’s change our code to connect to our `sample_restaurants` database and `restaurants` collection. We are going to be incorporating our aggregation pipeline to find a sample size of five American cuisine restaurants that are located in Brooklyn, New York. Let’s dive right in! 

Since we have our `pymongo` dependency downloaded, we can directly incorporate our aggregation pipeline into our code source. Change your `lambda_function.py` to look like this:

```
import os
from pymongo import MongoClient

connect = MongoClient(host=os.environ.get("ATLAS_URI"))

def lambda_handler(event, context):
    # Choose our "sample_restaurants" database and our "restaurants" collection
    database = connect.sample_restaurants
    collection = database.restaurants

    # This is our aggregation pipeline
    pipeline = [

        # We are finding American restaurants in Brooklyn
        {"$match": {"borough": "Brooklyn", "cuisine": "American"}},

        # We only want 5 out of our over 20k+ documents
        {"$limit": 5},

        # We don't want all the details, project what you need
        {"$project": {"_id": 0, "name": 1, "borough": 1, "cuisine": 1}}
        
    ]

    # This will show our pipeline 
    result = list(collection.aggregate(pipeline))

    # Print the result
    for restaurant in result:
        print(restaurant)
```
Here, we are using `$match` to find all the American cuisine restaurants located in Brooklyn. We are then using `$limit` to only five documents out of our database. Next, we are using `$project` to only show the fields we want. We are going to include “borough”, “cuisine”, and the “name” of the restaurant. Then, we are executing our pipeline and printing out our results. 

Click on “Deploy” to ensure our changes have been deployed to the code environment. After the changes are deployed, hit “Test.” We will get a sample size of five Brooklyn American restaurants as the result in our console:
![results from our aggregation pipeline shown in AWS Lambda](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta88b18f150f5e4bb/64a70e12ce1ee5779b8fc24f/Screenshot_2023-07-06_at_2.55.03_PM.png)

Our aggregation pipeline was successful!

## Conclusion

This tutorial provided you with hands-on experience to connect a MongoDB Atlas database to AWS Lambda. We also got an inside look on how to write to a cluster from Lambda, how to read back information from an aggregation pipeline, and how to properly configure our dependencies when using Lambda. Hopefully now, you are ready to take advantage of AWS Lambda and MongoDB to create the best applications without worrying about external infrastructure. 

If you enjoyed this tutorial and would like to learn more, please check out our [MongoDB Developer Center](https://www.mongodb.com/developer/) and [YouTube channel](https://www.youtube.com/@MongoDB).


Learn how to leverage the power of AWS Lambda and MongoDB Atlas in your applications. 

How to Use PyMongo to Connect MongoDB Atlas with AWS Lambda

For those who want to focus on creating scalable containerized applications without having to worry about managing any environments, this is the tutorial for you! We are going to be hosting a dockerized version of our previously built [Flask and MongoDB Atlas](https://www.mongodb.com/developer/languages/python/atlas-flask-and-azure-app-service/) application on [Azure Container Apps](https://learn.microsoft.com/en-us/azure/container-apps/overview).

Azure Container Apps truly simplifies not only the deployment but also the management of containerized applications and microservices on a serverless platform. This Microsoft service also offers a huge range of integrations with other Azure platforms, making it easy to scale or improve your application over time. The combination of Flask, [Atlas](https://www.mongodb.com/cloud/atlas/lp/try4?utm_source=google&utm_campaign=search_gs_pl_evergreen_atlas_general_prosp-brand_gic-null_amers-us_ps-all_desktop_eng_lead&utm_term=get%20started%20mongodb&utm_medium=cpc_paid_search&utm_ad=p&utm_ad_campaign_id=1718986498&adgroup=150907550594&cq_cmp=1718986498&gclid=CjwKCAjwov6hBhBsEiwAvrvN6H2IRATl9dAXll0-PVxWfdESK5f1bmHREd1o4WhKw-k6MjU0CtKYvBoCQdMQAvD_BwE), and Container Apps allows for developers to build applications that are capable of handling large amounts of data and traffic, while being extremely accessible from any machine or environment. 

The specifics of this tutorial are as follows: We will be cloning our previously built Flask application that utilizes CRUD (create, read, update, and delete) functionality applying to a “bookshelf” created in MongoDB Atlas. When properly up and running and connected to Postman or using cURL, we can add in new books, read back all the books in our database, update (exchange) a book, and even delete books. From here, we will dockerize our application and then we will host our dockerized image on Azure Container Apps. Once this is done, anyone anywhere can access our application!

The success of following this tutorial requires a handful of prerequisites:

* Access and clone our Flask and MongoDB application from the [GitHub repository](https://github.com/mongodb-developer/atlas-flask-azure-demo) if you would like to follow along.
* View the [completed repository for this demo](https://github.com/mongodb-developer/azurecontainerappdemo).
* [MongoDB Atlas](https://www.mongodb.com/atlas/database)
* [Docker Desktop](https://www.docker.com/products/docker-desktop/)
* [Microsoft Azure](https://azure.microsoft.com/en-us/free/search/?ef_id=_k_CjwKCAjwitShBhA6EiwAq3RqA3Ku-kj_rzgv0E-tGxmc8rI-IW7yDWeAvbLdQFM1WsZJaspuM49SeRoC1usQAvD_BwE_k_&OCID=AIDcmm5edswduu_SEM_k_CjwKCAjwitShBhA6EiwAq3RqA3Ku-kj_rzgv0E-tGxmc8rI-IW7yDWeAvbLdQFM1WsZJaspuM49SeRoC1usQAvD_BwE_k_&gclid=CjwKCAjwitShBhA6EiwAq3RqA3Ku-kj_rzgv0E-tGxmc8rI-IW7yDWeAvbLdQFM1WsZJaspuM49SeRoC1usQAvD_BwE) subscription.
* Python 3.9+.
* Postman Desktop (or another way to test our functions).

### Before we dive in...
Before we continue on to containerizing our application, please ensure you have a proper understanding of our program through this article: [Scaling for Demand: Deploying Python Applications Using MongoDB Atlas on Azure App Service](https://www.mongodb.com/developer/languages/python/atlas-flask-and-azure-app-service/). It goes into a lot of detail on how to properly build and connect a MongoDB Atlas database to our application along with the intricacies of the app itself. If you are a beginner, please ensure you completely understand the application prior to containerizing it through Docker. 

#### Insight into our database
Before moving on in our demo, if you’ve followed our previous demo linked above, this is how your Atlas database can look. These books were added in at the end of the previous demo using our endpoint. If you’re using your own application, an empty collection will be supported. But if you have existing documents, they need to support our schema or an error message will appear: 
![books in database](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt8640927974b6af94/6491be06c32681403e55e181/containerapps1.png)

We are starting with four novels with various pages. Once properly connected and hosted in Azure Container Apps, when we connect to our `/books` endpoint, these novels will show up. 

### Creating a Dockerfile

Once you have a cloned version of the application, it’s time to create our Dockerfile. A Dockerfile is important because it contains all of the information and commands to assemble an image. From the commands in a Dockerfile, Docker can actually build the image automatically with just one command from your CLI. 

In your working directory, create a new file called `Dockerfile` and put in these commands:

```
FROM python:3.9-slim-buster
WORKDIR /azurecontainerappsdemo


COPY ./config/requirements.txt /azurecontainerappsdemo/
RUN pip install -r requirements.txt


COPY . /azurecontainerappsdemo/


ENV FLASK_APP=app.py
EXPOSE 5000
CMD ["flask", "run", "--host=0.0.0.0"]
```

Please ensure your `requirements.txt` file is placed under a new folder called `/config`. This is so we can be certain our `requirements.txt` file is located and properly copied in with our Dockerfile since it is crucial for our demo.

Our base image `python:3.9-slim-buster` is essential because it provides a starting point for creating a new container image. In Docker, base images contain all the necessary components to successfully build and run an application. The rest of the commands copy over all the files in our working directory, expose Flask’s default port 5000, and specify how to run our application while allowing network access from anywhere. It is crucial to include the `--host=0.0.0.0` because otherwise, when we attempt to host our app on Azure, it will not connect properly.

In our app.py file, please make sure to add the following two lines at the very bottom of the file:
```
if __name__ == '__main__':
   app.run(host='0.0.0.0', debug=True)
```

This once again allows Flask to run the application, ensuring it is accessible from any network. 

###### Optional

You can test and make sure your app is properly dockerized with these commands:

Build: `docker build --tag azurecontainerappsdemo . `

Run: `docker run -d -p 5000:5000 -e "CONNECTION_STRING=<MONGODB_ATLAS_URI_STRING_HERE>" azurecontainerappsdemo`

You should see your app up and running on your local host.

![app hosted on local host](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta91a08c67a4a22c5/6491bea250d8ed2c592f2c2b/containerapps2.png)

Now, we can use Azure Container Apps to run our containerized application without worrying about infrastructure on a serverless platform. Let’s go over how to do this. 

### Creating an Azure Container Registry

We are going to be building and pushing our image to our Azure Container Registry in order to successfully host our application. To do this, please make sure that you are logged into Azure. There are multiple ways to create an Azure Container Registry: through the user interface, the command line, or even through the VSCode extension. For simplicity, this tutorial will show how to do it through the user interface. 

Our first step is to log into Azure and access the Container Registry service. Click to create a new registry and you will be taken to this page:
![how to create a container registry](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt04705caa092b7b97/6491da12359ef03a0860ef58/containerapps3.png)

Choose which Resource Group you want to use, along with a Registry Name (this will serve as your login URL) and Location. Make a note of these because when we start our Container App, all these need to be the same. After these are in place, press the Review and Create button. Once configured, your registry will look like this:

![container registry once created](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc9e46e8cf38d937b/6491da9c83c7fb0f375f56e2/containerapps4.png)

Now that you have your container registry in place, let’s access it in VSCode. Make sure that you have the Docker extension installed. Go to registries, log into your Azure account, and connect your registry. Mine is called “anaiyaregistry” and when set up, looks like this:
![Azure registry set up in VSCode](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd959e0e4e09566fb/6491db1d2429af7455f493d4/containerapps5.png)

Now, log into your ACR using this command: 
`docker login <azure registry url>`

As an example, mine is: 
`docker login anaiyaregistry.azurecr.io`

You will have to go to Access Keys inside of your Container Registry and click on Admin Access. Then, use that username and password to log into your terminal when prompted. If you are on a Windows machine, please make sure to right-click to paste. Otherwise, an error will appear:
![Access Keys page in our registry](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltfa57e9ac66da7827/6491db9e595392cc54a060bb/containerapps6.png)

When you’ve successfully logged in to your Azure subscription and Azure Registry, we can move on to building and pushing our image. 

### Building and pushing our image to Azure Container Registry

We need to now build our image and push it to our Azure Container Registry. 

If you are using an M1 Mac, we need to reconfigure our image so that it is using `amd64` instead of the configured `arm64`. This is because at the moment, Azure Container Apps only supports `linux/amd64` container images, and with an M1 machine, your image will automatically be built as `arm`. To get around this, we will be utilizing [Buildx](https://docs.docker.com/engine/reference/commandline/buildx/), a Docker plugin that allows you to build and push images for various platforms and architectures. 

If you are not using an M1 Mac, please skip to our “Non-M1 Machines” section.

#### Install Buildx
To install `buildx` on your machine, please put in the following commands:

`docker buildx install`

To enable `buildx` to use the Docker CLI, please type in:

`docker buildx create –use`

Once this runs and a randomized container name appears in your terminal, you’ll know `buildx` has been properly installed.

#### Building and pushing our image

The command to build our image is as follows:
`docker buildx build --platform linux/amd64 --t <azure registry url>/<image name>:<image tag> --output type=docker .`

As an example, my build command is: 
`docker buildx build --platform linux/amd64 --t anaiyaregistry.azurecr.io/azurecontainerappsdemo:latest --output type=docker .`

Specifying the platform you want your image to run on is the most important part. Otherwise, when we attempt to host it on Azure, we are going to get an error. 

Once this has succeeded, we need to push our image to our registry. We can do this with the command:

`docker push <azure registry url>/<image name>:<image tag>`

As an example, my push command is:

`docker push anaiyaregistry.azurecr.io/azurecontainerappsdemo:latest`

#### Non-M1 Mac machines
If you have a non-M1 machine, please follow the above steps but feel free to ignore installing `buildx`. For example, your build command will be:

`docker build --t <azure registry url>/<image name>:<image tag> --output type=docker .`

Your push command will be:

`docker push <azure registry url>/<image name>:<image tag>`

#### Windows machines
For Windows machines, please use the following build command:

`docker build --t <azure registry url>/<image name>:<image tag> .`

And use the following push command:

`docker push <image name>:<image tag>`

Once your push has been successful, let’s ensure we can properly see it in our Azure user interface. Access your Container Registries service and click on your registry. Then, click on Repositories. 
![where to find our repository on our Azure Container Registry’s page](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf60949e01ec5cba1/6491dc67359ef00b4960ef66/containerapps7.png)

Click again on your repository and there will be an image named `latest` since that is what we tagged our image with when we pushed it. This is the image we are going to host on our Container App service.

### Creating our Azure container app

We are going to be creating our container app through the Azure user interface. 

Access your Container Apps service and click Create. 

Now access your Container Apps in the UI. Click Create and fill in the “Basics” like this:
![how to create a container app in Azure Container Apps](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blted59e4c31f184cc4/6491dcd40f2d9b48bbed67a6/containerapps8.png)

**If this is the first time creating an Azure Container App, please ensure an [environment](https://learn.microsoft.com/en-us/azure/container-apps/environment) is created when the App is created. A Container Apps environment is crucial as it creates a secure boundary around various container apps that exist on the same virtual network. To check on your Container Apps environments, they can be accessed under “Container Apps Environments” in your Azure portal. 
![visual of our container app environment in Azure](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb71c98cdf43724e3/6491dd19ea50bc8939bea30e/containerapps9.png)

As we can see, the environment we chose from above is available under our Container Apps Environments tab. 

Please ensure your Region and Resource Group are identical to the options picked while creating your Registry in a previous step. Once you’re finished putting in the “Basics,” click App Settings and uncheck “Use quickstart image.” Under “Image Source,” click on Azure Container Registry and put in your image information.
![App Settings part of creating a Container App in Azure](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt38985f7d3ddeaa6a/6491dd718b23a55598054728/containerapps10.png)

At the bottom, enter your Environment Variable ([your connection string](https://www.mongodb.com/docs/guides/atlas/connection-string/). It’s located in both Atlas and your .env file, if copying over from the previous demo):
![how to properly set up our environment variables in Azure Container Apps](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7a9d4d12404bd90b/6491deb474d501e28e016236/containerapps11.png)

Under that, hit “Enabled” for ingress and fill in the rest like this:
![continuation of setting up our container app in Azure](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7b2b6dd5b0b8d228/6491def9ee654933dccceb84/containerapps12.png)

When done, hit Review and Create at the very bottom of the screen.

![Container App deployment page](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt6593fb8e678ddfcd/6491dfc0ea50bc4a92bea31f/containerapps13.png)

You’ll see this page when your deployment is successful. Hit Go to Resource.

![where to find our container app URL](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5c03d1267f75bd58/6491e006f7411b5c2137dd1a/containerapps14.png)

Click on the “Application URL” on the right-hand side. 
![our app hosted on Azure Container Apps](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt471796e40d6385b5/6491e04b0f2d9b22fded67c1/containerapps15.png)

You’ll be taken to your app.

If we change the URL to incorporate our ‘/books’ route, we will see all the books from our Atlas database! 
![viewing our books using our endpoint after hosting it on Azure Container Apps](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1fb382a03652147d/6491e074b9a076289492814a/containerapps16.png)

### Conclusion
Our Flask and MongoDB Atlas application has been successfully containerized and hosted on Azure Container Apps! Throughout this article, we’ve gone over how to create a Dockerfile and an Azure Container Registry, along with how to create and host our application on Azure Container Apps. 

Grab more details on [MongoDB Atlas](https://www.mongodb.com/atlas) or [Azure Container Apps](https://azure.microsoft.com/en-us/products/container-apps).


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt8640927974b6af94/6491be06c32681403e55e181/containerapps1.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta91a08c67a4a22c5/6491bea250d8ed2c592f2c2b/containerapps2.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt04705caa092b7b97/6491da12359ef03a0860ef58/containerapps3.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc9e46e8cf38d937b/6491da9c83c7fb0f375f56e2/containerapps4.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltd959e0e4e09566fb/6491db1d2429af7455f493d4/containerapps5.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltfa57e9ac66da7827/6491db9e595392cc54a060bb/containerapps6.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf60949e01ec5cba1/6491dc67359ef00b4960ef66/containerapps7.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blted59e4c31f184cc4/6491dcd40f2d9b48bbed67a6/containerapps8.png
  [9]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltb71c98cdf43724e3/6491dd19ea50bc8939bea30e/containerapps9.png
  [10]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt38985f7d3ddeaa6a/6491dd718b23a55598054728/containerapps10.png
  [11]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7a9d4d12404bd90b/6491deb474d501e28e016236/containerapps11.png
  [12]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7b2b6dd5b0b8d228/6491def9ee654933dccceb84/containerapps12.png
  [13]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt6593fb8e678ddfcd/6491dfc0ea50bc4a92bea31f/containerapps13.png
  [14]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5c03d1267f75bd58/6491e006f7411b5c2137dd1a/containerapps14.png
  [15]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt471796e40d6385b5/6491e04b0f2d9b22fded67c1/containerapps15.png
  [16]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt1fb382a03652147d/6491e074b9a076289492814a/containerapps16.png

This tutorial explains how to host your MongoDB Atlas application on Azure Container Apps for a scalable containerized solution.


Building a Flask and MongoDB App with Azure Container Apps

Nic Raboy

In the developer community, ensuring your projects run accurately regardless of the environment can be a pain. Whether it’s trying to recreate a demo from an online tutorial or working on a code review, hearing the words, “Well, it works on my machine…” can be frustrating. Instead of spending hours debugging, we want to introduce you to a platform that will change your developer experience: [Docker](https://www.docker.com/). 

Docker is a great tool to learn because it provides developers with the ability for their applications to be used easily between environments, and it's resource-efficient in comparison to virtual machines. This tutorial will gently guide you through how to navigate Docker, along with how to integrate [Go](https://golang.org/) on the platform. We will be using this project to connect to our previously built MongoDB Atlas Search Cluster made for using [Synonyms in Atlas Search](https://www.mongodb.com/developer/how-to/get-hyped-synonyms-atlas-search/). Stay tuned for a fun read on how to learn all the above while also expanding your Gen-Z slang knowledge from our synonyms cluster. Get hyped! 

## The Prerequisites

There are a few requirements that must be met to be successful with this tutorial.

- A M0 or better MongoDB Atlas cluster
- Docker Desktop

To use MongoDB with the Golang driver, you only need a free M0 cluster. To create this cluster, follow the instructions listed on the [MongoDB documentation](https://docs.atlas.mongodb.com/tutorial/create-new-cluster/). However, we’ll be making many references to a [previous tutorial](https://www.mongodb.com/developer/how-to/get-hyped-synonyms-atlas-search/) where we used Atlas Search with custom synonyms.

Since this is a Docker tutorial, you’ll need [Docker Desktop](https://www.docker.com/products/docker-desktop). You don’t actually need to have Golang configured on your host machine because Docker can take care of this for us as we progress through the tutorial.

## Building a Go API with the MongoDB Golang Driver 

Like previously mentioned, you don’t need Go installed and configured on your host computer to be successful. However, it wouldn’t hurt to have it in case you wanted to test things prior to creating a Docker image.

On your computer, create a new project directory, and within that project directory, create a **src** directory with the following files:

- go.mod
- main.go

The **go.mod** file is our dependency management file for Go modules. It could easily be created manually or by using the following command:

```bash
go mod init
```

The **main.go** file is where we’ll keep all of our project code.

Starting with the **go.mod** file, add the following lines:

```
module github.com/mongodb-developer/docker-golang-example
go 1.15
require go.mongodb.org/mongo-driver v1.7.0
require github.com/gorilla/mux v1.8.0
```

Essentially, we’re defining what version of Go to use and the modules that we want to use. For this project, we’ll be using the MongoDB Go driver as well as the [Gorilla Web Toolkit](https://www.gorillatoolkit.org/).

This brings us into the building of our simple API.

Within the **main.go** file, add the following code:

```golang
package main

import (
	"context"
	"encoding/json"
	"fmt"
	"net/http"
	"os"
	"time"

	"github.com/gorilla/mux"
	"go.mongodb.org/mongo-driver/bson"
	"go.mongodb.org/mongo-driver/mongo"
	"go.mongodb.org/mongo-driver/mongo/options"
)

var client *mongo.Client
var collection *mongo.Collection

type Tweet struct {
	ID       int64  `json:"_id,omitempty" bson:"_id,omitempty"`
	FullText string `json:"full_text,omitempty" bson:"full_text,omitempty"`
	User     struct {
		ScreenName string `json:"screen_name" bson:"screen_name"`
	} `json:"user,omitempty" bson:"user,omitempty"`
}

func GetTweetsEndpoint(response http.ResponseWriter, request *http.Request) {}
func SearchTweetsEndpoint(response http.ResponseWriter, request *http.Request) {}

func main() {
	fmt.Println("Starting the application...")
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer cancel()
	client, err := mongo.Connect(ctx, options.Client().ApplyURI(os.Getenv("MONGODB_URI")))
	defer func() {
		if err = client.Disconnect(ctx); err != nil {
			panic(err)
		}
	}()
collection = client.Database("synonyms").Collection("tweets")
	router := mux.NewRouter()
	router.HandleFunc("/tweets", GetTweetsEndpoint).Methods("GET")
	router.HandleFunc("/search", SearchTweetsEndpoint).Methods("GET")
	http.ListenAndServe(":12345", router)
}
```

There’s more to the code, but before we see the rest, let’s start breaking down what we have above to make sense of it.

You’ll probably notice our `Tweets` data structure:

```golang
type Tweet struct {
	ID       int64  `json:"_id,omitempty" bson:"_id,omitempty"`
	FullText string `json:"full_text,omitempty" bson:"full_text,omitempty"`
	User     struct {
		ScreenName string `json:"screen_name" bson:"screen_name"`
	} `json:"user,omitempty" bson:"user,omitempty"`
}
```

Earlier in the tutorial, we mentioned that this example is heavily influenced by a [previous tutorial](https://www.mongodb.com/developer/how-to/get-hyped-synonyms-atlas-search/) that used Twitter data. We highly recommend you take a look at it. This data structure has some of the fields that represent a tweet that we scraped from Twitter. We didn’t map all the fields because it just wasn’t necessary for this example.

Next, you’ll notice the following:

```golang
func GetTweetsEndpoint(response http.ResponseWriter, request *http.Request) {}
func SearchTweetsEndpoint(response http.ResponseWriter, request *http.Request) {}
```

These will be the functions that hold our API endpoint logic. We’re going to skip these for now and focus on understanding the connection and configuration logic.

As of now, most of what we’re interested in is happening in the `main` function.

The first thing we’re doing is connecting to MongoDB:

```golang
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI(os.Getenv("MONGODB_URI")))
defer func() {
	if err = client.Disconnect(ctx); err != nil {
		panic(err)
	}
}()
collection = client.Database("synonyms").Collection("tweets")
```

You’ll probably notice the `MONGODB_URI` environment variable in the above code. It’s not a good idea to hard-code the MongoDB connection string in the application. This prevents us from being flexible and it could be a security risk. Instead, we’re using environment variables that we’ll pass in with Docker when we deploy our containers.

You can visit the MongoDB Atlas dashboard for your URI string.

The database we plan to use is `synonyms` and we plan to use the `tweets` collection, both of which we talked about in that previous tutorial.

After connecting to MongoDB, we focus on configuring the Gorilla Web Toolkit:

```golang
router := mux.NewRouter()
router.HandleFunc("/tweets", GetTweetsEndpoint).Methods("GET")
router.HandleFunc("/search", SearchTweetsEndpoint).Methods("GET")
http.ListenAndServe(":12345", router)
```

In this code, we are defining which endpoint path should route to which function. The functions are defined, but we haven’t yet added any logic to them. The application itself will be serving on port 12345.

As of now, the application has the necessary basic connection and configuration information. Let’s circle back to each of those endpoint functions.

We’ll start with the `GetTweetsEndpoint` because it will work fine with an M0 cluster:

```golang
func GetTweetsEndpoint(response http.ResponseWriter, request *http.Request) {
	response.Header().Set("content-type", "application/json")
	var tweets []Tweet
	ctx, _ := context.WithTimeout(context.Background(), 30*time.Second)
	cursor, err := collection.Find(ctx, bson.M{})
	if err != nil {
		response.WriteHeader(http.StatusInternalServerError)
		response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
		return
	}
	if err = cursor.All(ctx, &tweets); err != nil {
		response.WriteHeader(http.StatusInternalServerError)
		response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
		return
	}
	json.NewEncoder(response).Encode(tweets)
}
```

In the above code, we’re saying that we want to use the `Find` operation on our collection for all documents in that collection, hence the empty filter object.

If there were no errors, we can get all the results from our cursor, load them into a `Tweet` slice, and then JSON encode that slice for sending to the client. The client will receive JSON data as a result.

Now we can look at the more interesting endpoint function.

```golang
func SearchTweetsEndpoint(response http.ResponseWriter, request *http.Request) {
	response.Header().Set("content-type", "application/json")
	queryParams := request.URL.Query()
	var tweets []Tweet
	ctx, _ := context.WithTimeout(context.Background(), 30*time.Second)
	searchStage := bson.D{
		{"$search", bson.D{
			{"index", "synsearch"},
			{"text", bson.D{
				{"query", queryParams.Get("q")},
				{"path", "full_text"},
				{"synonyms", "slang"},
			}},
		}},
	}
	cursor, err := collection.Aggregate(ctx, mongo.Pipeline{searchStage})
	if err != nil {
		response.WriteHeader(http.StatusInternalServerError)
		response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
		return
	}
	if err = cursor.All(ctx, &tweets); err != nil {
		response.WriteHeader(http.StatusInternalServerError)
		response.Write([]byte(`{ "message": "` + err.Error() + `" }`))
		return
	}
	json.NewEncoder(response).Encode(tweets)
}
```

The idea behind the above function is that we want to use an aggregation pipeline for Atlas Search. It does use the synonym information that we outlined in the [previous tutorial](https://www.mongodb.com/developer/how-to/get-hyped-synonyms-atlas-search/).

The first important thing in the above code to note is the following:

```golang
​​queryParams := request.URL.Query()
```

We’re obtaining the query parameters passed with the HTTP request. We’re expecting a `q` parameter to exist with the search query to be used.

To keep things simple, we make use of a single stage for the MongoDB aggregation pipeline:

```golang
searchStage := bson.D{
	{"$search", bson.D{
		{"index", "synsearch"},
		{"text", bson.D{
			{"query", queryParams.Get("q")},
			{"path", "full_text"},
			{"synonyms", "slang"},
		}},
	}},
}
```

In this stage, we are doing a text search with a specific index and a specific set of synonyms. The query that we use for our text search comes from the query parameter of our HTTP request.

Assuming that everything went well, we can load all the results from the cursor into a `Tweet` slice, JSON encode it, and return it to the client that requested it.

If you have Go installed and configured on your computer, go ahead and try to run this application. Just don’t forget to add the `MONGODB_URI` to your environment variables prior.

If you want to learn more about API development with the Gorilla Web Toolkit and MongoDB, check out [this tutorial](https://www.thepolyglotdeveloper.com/2019/02/developing-restful-api-golang-mongodb-nosql-database/) on the subject.

## Configuring a Docker Image for Go with MongoDB

Let’s get started with Docker! If it’s a platform you’ve never used before, it might seem a bit daunting at first, but let us guide you through it, step by step. We will be showing you how to download Docker and get started with setting up your first Dockerfile to connect to our Gen-Z Synonyms Atlas Cluster. 

First things first. Let’s download Docker. This can be done through their [website](https://www.docker.com/products/docker-desktop) in just a couple of minutes. 

Once you have that up and running, it’s time to create your very first Dockerfile. 

At the root of your project folder, create a new **Dockerfile** file with the following content:

```
#get a base image
FROM golang:1.16-buster

MAINTAINER anaiya raisinghani <anaiya.raisinghani@mongodb.com>

WORKDIR /go/src/app
COPY ./src .

RUN go get -d -v
RUN go build -v

CMD ["./docker-golang-example"]
```

This format is what many Dockerfiles are composed of, and a lot of it is heavily customizable and can be edited to fit your project's needs. 

The first step is to grab a base image that you’re going to use to build your new image. You can think of using Dockerfiles as layers to a cake. There are a multitude of different base images out there, or you can use `FROM scratch` to start from an entirely blank image. Since this project is using the programming language Go, we chose to start from the `golang` base image and add the tag `1.16` to represent the version of Go that we plan to use. Whenever you include a tag next to your base image, be sure to set it up with a colon in between, just like this: `golang:1.16`. To learn more about which tag will benefit your project the best, check out [Docker’s documentation](https://hub.docker.com/_/golang?tab=tags&page=1&ordering=last_updated) on the subject.

This site holds a lot of different tags that can be used on a Golang base image. Tags are important because they hold very valuable information about the base image you’re using such as software versions, operating system flavor, etc. 

Let’s run through the rest of what will happen in this Dockerfile!

It's optional to include a `MAINTAINER` for your image, but it’s good practice so that people viewing your Dockerfile can know who created it. It's not necessary, but it’s helpful to include your full name and your email address in the file. 

The `WORKDIR /go/src/app` command is crucial to include in your Dockerfile since `WORKDIR` specifies which working directory you’re in. All the commands after will be run through whichever directory you choose, so be sure to be aware of which directory you’re currently in.

The `COPY ./src .` command allows you to copy whichever files you want from the specified location on the host machine into the Docker image. 

Now, we can use the `RUN` command to set up exactly what we want to happen at image build time before deploying as a container. The first command we have is `RUN go get -d -v`, which will download all of the Go dependencies listed in the **go.mod** file that was copied into the image.. 

Our second `RUN` command is `RUN go build -v`, which will build our project into an executable binary file. 

The last step of this Dockerfile is to use a `CMD` command, `CMD [“./docker-golang-example”]`. This command will define what is run when the container is deployed rather than when the image is built. Essentially we’re saying that we want the built Go application to be run when the container is deployed.

Once you have this Dockerfile set up, you can build and execute your project using your entire MongoDB URI link:

To build the Docker image and deploy the container, execute the following from the command line:

```bash
docker build -t docker-syn-image .
docker run -d -p 12345:12345 -e “MONGODB_URI=YOUR_URI_HERE” docker-syn-image
```

Following these instructions will allow you to run the project and access it from http://localhost:12345. **But**! It’s so tedious. What if we told you there was an easier way to run your application without having to write in the entire URI link? There is! All it takes is one extra step: setting up a Docker Compose file. 

## Setting Up a Docker Compose File to Streamline Deployments

A Docker Compose file is a nice little step to run all your container files and dependencies through a simple command: `docker compose up`.

In order to set up this file, you need to establish a YAML configuration file first. Do this by creating a new file in the root of your project folder, naming it **docker-compose**, and adding **.yml** at the end. You can name it something else if you like, but this is the easiest since when running the `docker compose up` command, you won’t need to specify a file name. Once that is in your project folder, follow the steps below.

This is what your Docker Compose file will look like once you have it all set up: 

```yaml
version: "3.9" 
services:
   web:
       build: .
       ports:
           - "12345:12345"
       environment:
           MONGODB_URI: your_URI_here
```

Let’s run through it!

First things first. Determine which schema version you want to be running. You should be using the most recent version, and you can find this out through [Docker’s documentation](https://docs.docker.com/compose/compose-file/).

Next, define which services, otherwise known as containers, you want to be running in your project. We have included `web` since we are attaching to our Atlas Search cluster. The name isn’t important and it acts more as an identifier for that particular service. Next, specify that you are building your application, and put in your `ports` information in the correct spot. For the next step, we can set up our `environment` as our MongoDB URI and we’re done! 

Now, run the command `docker compose up` and watch the magic happen. Your container should build, then run, and you’ll be able to connect to your port and see all the tweets!

## Conclusion

This tutorial has now left you equipped with the knowledge you need to build a Go API with the MongoDB Golang driver, create a Dockerfile, create a Docker Compose file, and connect your newly built container to a MongoDB Atlas Cluster. 

Using these new platforms will allow you to take your projects to a whole new level. 

If you’d like to take a look at the code used in our project, you can access it on [GitHub](https://github.com/mongodb-developer/docker-golang-example).

Using Docker or Go, but have a question? Check out the [MongoDB Community Forums](https://community.mongodb.com)!

Learn how to create and deploy Golang-powered micro-services that interact with MongoDB using Docker.

Get Hyped: Using Docker + Go with MongoDB

Michael Lynn

Douglas Eck is a Principal Scientist at Google Research and a research director on the Brain Team. His work lies at the intersection of machine learning and human-computer interaction (HCI). Doug created and helps lead Magenta (g.co/magenta), an ongoing research project exploring the role of machine learning in the process of creating art and music. This article is a transcript of the podcast episode where Anaiya Rasinghani leads an interview with Doug to learn more about the intersection between AI, ML, HCI, and Databases.

At the Intersection of AI/ML and HCI with Douglas Eck of Google (MongoDB Podcast)

Sometimes, the word you’re looking for is on the tip of your tongue, but you can’t quite grasp it. For example, when you’re  trying to find a really funny tweet you saw last night to show your friends. If you’re sitting there reading this and thinking, "Wow, Anaiya and Nic, you’re so right. I wish there was a fix for this," strap on in! We have just the solution for those days when your precise linguistic abilities fail you, but you have an idea of what you’re looking for: **Synonyms in Atlas Search**. 

In this tutorial, we are going to be showing you how to index a MongoDB collection to capture searches for words that mean similar things. For the specifics, we’re going to search through content written with Generation Z (Gen-Z) slang. The slang will be mapped to common words with synonyms and as a result, you’ll get a quick Gen-Z lesson without having to ever open TikTok. 

If you’re in the mood to learn a few new words, alongside how effortlessly synonym mappings can be integrated into Atlas Search, this is the tutorial for you. 

## Requirements

There are a few requirements that must be met to be successful with this tutorial:

- MongoDB Atlas M0 (or higher) cluster running MongoDB version 4.4 (or higher)
- Node.js
- A Twitter developer account

We’ll be using Node.js to load our Twitter data, but a Twitter developer account is required for accessing the APIs that contain Tweets.

## Load Twitter Data into a MongoDB Collection

![Example Tweet Data for Slang Synonyms](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/tweets_7ac7bd49a0.png "Example Tweet Data for Slang Synonyms")

Before starting this section of the tutorial, you’re going to need to have your Twitter API Key and API Secret handy. These can both be generated from the [Twitter Developer Portal](https://developer.twitter.com/en/portal/petition/use-case).

The idea is that we want to store a bunch of tweets in MongoDB that contain Gen-Z slang that we can later make sense of using Atlas Search and properly defined synonyms. Each tweet will be stored as a single document within MongoDB and will look something like this:

```json
{
    "_id": 1420091624621629400,
    "created_at": "Tue Jul 27 18:40:01 +0000 2021",
    "id": 1420091624621629400,
    "id_str": "1420091624621629443",
    "full_text": "Don't settle for a cheugy database, choose MongoDB instead 💪",
    "truncated": false,
    "entities": {
        "hashtags": [],
        "symbols": [],
        "user_mentions": [],
        "urls": []
    },
    "metadata": {
        "iso_language_code": "en",
        "result_type": "recent"
    },
    "source": "<a href=\"https://mobile.twitter.com\" rel=\"nofollow\">Twitter Web App</a>",
    "in_reply_to_status_id": null,
    "in_reply_to_status_id_str": null,
    "in_reply_to_user_id": null,
    "in_reply_to_user_id_str": null,
    "in_reply_to_screen_name": null,
    "user": {
        "id": 1400935623238643700,
        "id_str": "1400935623238643716",
        "name": "Anaiya Raisinghani",
        "screen_name": "anaiyaraisin",
        "location": "",
        "description": "Developer Advocacy Intern @MongoDB. Opinions are my own!",
        "url": null,
        "entities": {
            "description": {
                "urls": []
            }
        },
        "protected": false,
        "followers_count": 11,
        "friends_count": 29,
        "listed_count": 1,
        "created_at": "Fri Jun 04 22:01:07 +0000 2021",
        "favourites_count": 8,
        "utc_offset": null,
        "time_zone": null,
        "geo_enabled": false,
        "verified": false,
        "statuses_count": 7,
        "lang": null,
        "contributors_enabled": false,
        "is_translator": false,
        "is_translation_enabled": false,
        "profile_background_color": "F5F8FA",
        "profile_background_image_url": null,
        "profile_background_image_url_https": null,
        "profile_background_tile": false,
        "profile_image_url": "http://pbs.twimg.com/profile_images/1400935746593202176/-pgS_IUo_normal.jpg",
        "profile_image_url_https": "https://pbs.twimg.com/profile_images/1400935746593202176/-pgS_IUo_normal.jpg",
        "profile_banner_url": "https://pbs.twimg.com/profile_banners/1400935623238643716/1622845231",
        "profile_link_color": "1DA1F2",
        "profile_sidebar_border_color": "C0DEED",
        "profile_sidebar_fill_color": "DDEEF6",
        "profile_text_color": "333333",
        "profile_use_background_image": true,
        "has_extended_profile": true,
        "default_profile": true,
        "default_profile_image": false,
        "following": null,
        "follow_request_sent": null,
        "notifications": null,
        "translator_type": "none",
        "withheld_in_countries": []
    },
    "geo": null,
    "coordinates": null,
    "place": null,
    "contributors": null,
    "is_quote_status": false,
    "retweet_count": 0,
    "favorite_count": 1,
    "favorited": false,
    "retweeted": false,
    "lang": "en"
}
```

The above document model is more extravagant than we need. In reality, we’re only going to be paying attention to the `full_text` field, but it’s still useful to know what exists for any given tweet.

Now that we know what the document model is going to look like, we just need to consume it from Twitter.

We’re going to use two different Twitter APIs with our API Key and API Secret. The first API is the authentication API and it will give us our access token. With the access token we can get tweet data based on a Twitter query.

Since we’re using Node.js, we need to install our dependencies. Within a new directory on your computer, execute the following commands from the command line:

```bash
npm init -y
npm install mongodb axios --save
```

The above commands will create a new **package.json** file and install the MongoDB Node.js driver as well as Axios for making HTTP requests.

Take a look at the following Node.js code which can be added to a **main.js** file within your project:

```javascript
const { MongoClient } = require("mongodb");
const axios = require("axios");

require("dotenv").config();

const mongoClient = new MongoClient(process.env.MONGODB_URI);

(async () => {
    try {
        await mongoClient.connect();
        const tokenResponse = await axios({
            "method": "POST",
            "url": "https://api.twitter.com/oauth2/token",
            "headers": {
                "Authorization": "Basic " + Buffer.from(`${process.env.API_KEY}:${process.env.API_SECRET}`).toString("base64"),
                "Content-Type": "application/x-www-form-urlencoded"
            },
            "data": "grant_type=client_credentials"
        });
        const tweetResponse = await axios({
            "method": "GET",
            "url": "https://api.twitter.com/1.1/search/tweets.json",
            "headers": {
                "Authorization": "Bearer " + tokenResponse.data.access_token
            },
            "params": {
                "q": "mongodb -filter:retweets filter:safe (from:codeSTACKr OR from:nraboy OR from:kukicado OR from:judy2k OR from:adriennetacke OR from:anaiyaraisin OR from:lauren_schaefer)",
                "lang": "en",
                "count": 100,
                "tweet_mode": "extended"
            }
        });
        console.log(`Next Results: ${tweetResponse.data.search_metadata.next_results}`)
        const collection = mongoClient.db(process.env.MONGODB_DATABASE).collection(process.env.MONGODB_COLLECTION);
        tweetResponse.data.statuses = tweetResponse.data.statuses.map(status => {
            status._id = status.id;
            return status;
        });
        const result = await collection.insertMany(tweetResponse.data.statuses);
        console.log(result);
    } finally {
        await mongoClient.close();
    }
})();
```

There’s quite a bit happening in the above code so we’re going to break it down. However, before we break it down, it's important to note that we’re using environment variables for a lot of the sensitive information like tokens, usernames, and passwords. For security reasons, you really shouldn’t hard-code these values.

Inside the asynchronous function, we attempt to establish a connection to MongoDB. If successful, no error is thrown, and we make our first HTTP request.

```javascript
const tokenResponse = await axios({
    "method": "POST",
    "url": "https://api.twitter.com/oauth2/token",
    "headers": {
        "Authorization": "Basic " + Buffer.from(`${process.env.API_KEY}:${process.env.API_SECRET}`).toString("base64"),
        "Content-Type": "application/x-www-form-urlencoded"
    },
    "data": "grant_type=client_credentials"
});
```

Once again, in this first HTTP request, we are exchanging our API Key and API Secret with an access token to be used in future requests.

Using the access token from the response, we can make our second request to the tweets API endpoint:

```javascript
const tweetResponse = await axios({
    "method": "GET",
    "url": "https://api.twitter.com/1.1/search/tweets.json",
    "headers": {
        "Authorization": "Bearer " + tokenResponse.data.access_token
    },
    "params": {
        "q": "mongodb -filter:retweets filter:safe",
        "lang": "en",
        "count": 100,
        "tweet_mode": "extended"
    }
});
```

The tweets API endpoint expects a Twitter specific query and some other optional parameters like the language of the tweets or the expected result count. You can check the query language in the [Twitter documentation](https://developer.twitter.com/en/docs/twitter-api/v1/rules-and-filtering/search-operators).

At this point, we have an array of tweets to work with.

The next step is to pick the database and collection we plan to use and insert the array of tweets as documents. We can use a simple `insertMany` operation like this:

```javascript
const result = await collection.insertMany(tweetResponse.data.statuses);
```

The `insertMany` takes an array of objects, which we already have. We have an array of tweets, so each tweet will be inserted as a new document within the database.

If you have the MongoDB shell handy, you can validate the data that was inserted by executing the following:

```javascript
use("synonyms");
db.tweets.find({ });
```

Now that there’s data to work with, we can start to search it using slang synonyms.

## Creating Synonym Mappings in MongoDB 

While we’re using a `tweets` collection for our actual searchable data, the synonym information needs to exist in a separate source collection in the same database. 

You have two options for how you want your synonyms to be mapped–explicit or equivalent. You are not stuck with choosing just one type. You can have a combination of both explicit and equivalent as synonym documents in your collection. Choose the explicit format for when you need a set of terms to show up as a result of your inputted term, and choose equivalent if you want all terms to show up bidirectionally regardless of your queried term. 

For example, the word "basic" means "regular" or "boring." If we decide on an explicit (one-way) mapping for "basic," we are telling Atlas Search that if someone searches for "basic," we want to return all documents that include the words "basic," "regular," and "boring." But! If we query the word "regular," we would not get any documents that include "basic" because "regular" is not explicitly mapped to "basic." 

If we decide to map "basic" equivalently to "regular" and "boring," whenever we query any of these words, all the documents containing "basic," "regular," **and** "boring" will show up regardless of the initial queried word. 

To learn more about explicit vs. equivalent synonym mappings, check out the [official documentation](https://docs.atlas.mongodb.com/reference/atlas-search/synonyms/). 

For our demo, we decided to make all of our synonyms equivalent and formatted our synonym data like this: 

```json
[
    {
        "mappingType": "equivalent",
        "synonyms": ["basic", "regular", "boring"]  
    },
    {
        "mappingType": "equivalent",
        "synonyms": ["bet", "agree", "concur"]
    },
    {
        "mappingType": "equivalent",
        "synonyms": ["yikes", "embarrassing", "bad", "awkward"]
    },
    {
        "mappingType": "equivalent",
        "synonyms": ["fam", "family", "friends"]
    }
]
```

Each object in the above array will exist as a separate document within MongoDB. Each of these documents contains information for a particular set of synonyms.

To insert your synonym documents into your MongoDB collection, you can use the ‘insertMany()’ MongoDB raw function to put all your documents into the collection of your choice. 

```javascript
use("synonyms");

db.slang.insertMany([
    {
        "mappingType": "equivalent",
        "synonyms": ["basic", "regular", "boring"]
    },
    {
        "mappingType": "equivalent",
        "synonyms": ["bet", "agree", "concur"]
    }
]);
```

The `use("synonyms");` line is to ensure you’re in the correct database before inserting your documents. We’re using the `slang` collection to store our synonyms and it doesn’t need to exist in our database prior to running our query.

## Create an Atlas Search Index that Leverages Synonyms

Once you have your collection of synonyms handy and uploaded, it's time to create your search index! A search index is crucial because it allows you to use full-text search to find the inputted queries in that collection. 

We have included screenshots below of what your MongoDB Atlas Search user interface will look like so you can follow along: 

The first step is to click on the "Search" tab, located on your cluster page in between the "Collections" and "Profiler" tabs.

![Find the Atlas Search Tab](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/atlas_search_tab_830c2159ce.png "Find the Atlas Search Tab")

The second step is to click on the "Create Index" button in the upper right hand corner, or if this is your first Index, it will be located in the middle of the page. 

![Create a New Atlas Search Index](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_atlas_search_index_e03316726a.png "Create a New Atlas Search Index")

Once you reach this page, go ahead and click "Next" and continue on to the page where you will name your Index and set it all up! 

![Name the Atlas Search Index](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/name_atlas_search_index_e8284eb612.png "Name the Atlas Search Index")

Click "Next" and you’ll be able to create your very own search index! 

![Finalize the Atlas Search Index](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/finalize_atlas_search_index_b475015a58.png "Finalize the Atlas Search Index")

Once you create your search index, you can go back into it and then edit your index definition using the JSON editor to include what you need. The index we wrote for this tutorial is below: 

```json
{
    "mappings": {
        "dynamic": true
    },
    "synonyms": [
        {
            "analyzer": "lucene.standard",
            "name": "slang",
            "source": {
                "collection": "slang"
            }
        }
    ]
}
```

Let’s run through this! 

```json
{
    "mappings": {
    "dynamic": true
},
```

You have the option of choosing between dynamic and static for your search index, and this can be up to your discretion. To find more information on the difference between dynamic and static mappings, check out the [documentation](https://docs.atlas.mongodb.com/reference/atlas-search/synonyms/).

```json
"synonyms": [
    {
        "analyzer": "lucene.standard",
        "name": "slang",
        "source": {
            "collection": "slang"
        }
    }
]
```

This section refers to the synonyms associated with the search index. In this example, we’re giving this synonym mapping a name of "slang," and we’re using the default index analyzer on the synonym data, which can be found in the slang collection. 

## Searching with Synonyms with the MongoDB Aggregation Pipeline

Our next step is to put together the search query that will actually filter through your tweet collection and find the tweets you want using synonyms! 

The code we used for this part is below:

```javascript
use("synonyms");

db.tweets.aggregate([
   {
       "$search": {
           "index": "synsearch",
           "text": {
               "query": "throw",
               "path": "full_text",
               "synonyms": "slang"
           }
       }
   }
]);
```

We want to search through our tweets and find the documents containing synonyms for our query "throw." This is the synonym document for "throw":

```json
{
    "mappingType": "equivalent",
    "synonyms": ["yeet", "throw", "agree"]
},
```

Remember to include the name of your search index from earlier (synsearch). Then, the query we’re specifying is "throw." This means we want to see tweets that include "yeet," "throw," and "agree" once we run this script. 

The ‘path’ represents the field we want to search within, and in this case, we are searching for "throw" only within the ‘full_text’ field of the documents and no other field. Last but not least, we want to use synonyms found in the collection we have named "slang." 

Based on this query, any matches found will include the entire document in the result-set. To better streamline this, we can use a `$project` aggregation stage to specify the fields we’re interested in. This transforms our query into the following aggregation pipeline:

```javascript
db.tweets.aggregate([
    {
        "$search": {
            "index": "synsearch",
            "text": {
                "query": "throw",
                "path": "full_text",
                "synonyms": "slang"
            }
        }
    },
    {
        "$project": {
            "_id": 1,
            "full_text": 1,
            "username": "$user.screen_name"
        }
    }
]);
```

And these are our results! 

```json
[
    {
        "_id": 1420084484922347500,
        "full_text": "not to throw shade on SQL databases, but MongoDB SLAPS",
        "username": "codeSTACKr"
    },
    {
        "_id": 1420088203499884500,
        "full_text": "Yeet all your data into a MongoDB collection and watch the magic happen! No cap, we are efficient 💪",
        "username": "nraboy"
    }
]
```

Just as we wanted, we have tweets that include the word "throw" and the word "yeet!" 

## Conclusion

We’ve accomplished a **ton** in this tutorial, and we hope you’ve enjoyed following along. Now, you are set with the knowledge to load in data from external sources, create your list of explicit or equivalent synonyms and insert it into a collection, and write your own index search script. Synonyms can be useful in a multitude of ways, not just isolated to Gen-Z slang. From figuring out regional variations (e.g., soda = pop), to finding typos that cannot be easily caught with autocomplete, incorporating synonyms will help save you time and a thesaurus. 

Using synonyms in Atlas Search will improve your app’s search functionality and will allow you to find the data you’re looking for, even when you can’t quite put your finger on it. 

If you want to take a look at the code, queries, and indexes used in this blog post, check out the project on [GitHub](https://github.com/mongodb-developer/twitter-slang-search). If you want to learn more about synonyms in Atlas Search, check out the [documentation](https://docs.atlas.mongodb.com/reference/atlas-search/synonyms/).

If you have questions, please head to our [developer community website](https://www.mongodb.com/community/forums/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

Learn how to define your own custom synonyms for use with MongoDB Atlas Search in this example with features searching within slang found in Twitter messages.

Get Hyped: Synonyms in Atlas Search

This tutorial will show you how to create a functional Flask application that connects to a MongoDB Atlas database and is hosted on Azure App Service.