Ashwin Gangadhar

The daunting task that developers currently face while developing solutions powered by the retrieval augmented generation (RAG) framework is the choice of retrieval mechanism. Augmenting the large language model (LLM) prompt with relevant and exhaustive information creates better responses from such systems. One is tasked with choosing the most appropriate embedding model in the case of semantic similarity search. Alternatively, in the case of full-text search implementation, you have to be thorough about your implementation to achieve a precise recall and high accuracy in your results. Sometimes, the solutions require a combined implementation that benefits from both retrieval mechanisms.

If your current full-text search scoring workflow is leaving things to be desired, or if you find yourself spending too much time writing numerous lines of code to get semantic search functionality working within your applications, then Cohere and MongoDB can help. To prevent these issues from holding you back from leveraging powerful AI search functionality or machine learning within your application, Cohere and MongoDB offer easy-to-use and fully managed solutions.

[Cohere](https://cohere.ai/) is an AI company specializing in [large language models](https://www.forbes.com/sites/garydrenik/2023/01/11/large-language-models-will-define-artificial-intelligence/?sh=472a9243b60f).

1. With a powerful tool for embedding natural language in their projects, it can help you represent more accurate, relevant, and engaging content as embeddings.[ The Cohere language model](https://docs.cohere.com/docs/introduction-to-large-language-models) also offers a simple and intuitive API that allows you to easily integrate it with your existing workflows and platforms. 
2. The [Cohere Rerank](https://cohere.com/rerank) module is a component of the Cohere natural language processing system that helps to select the best output from a set of candidates. The module uses a neural network to score each candidate based on its relevance, semantic similarity, theme, and style. The module then ranks the candidates according to their scores and returns the top N as the final output.

MongoDB Atlas is a fully managed modern database service that provides scalable, secure, and reliable data storage and access for your applications. One of the key features of MongoDB Atlas is the ability to perform vector search and full-text search on your data, which can enhance the capabilities of your AI/ML-driven applications. MongoDB Atlas can help you build powerful and flexible AI/ML-powered applications that can leverage both structured and unstructured data. You can easily create and manage search indexes, perform queries, and analyze results using MongoDB Atlas's intuitive interface, APIs, and drivers. MongoDB Atlas Vector Search provides a unique feature — pre-filtering and post-filtering on vector search queries — that helps users control the behavior of their vector search results, thereby improving the accuracy and retrieval performance, and saving money at the same time.

Therefore, with Cohere and MongoDB Atlas, we can demonstrate techniques where we can easily power a semantic search capability on your private dataset with very few lines of code. Additionally, you can enhance the existing ranking of your full-text search retrieval systems using the Cohere Rerank module. Both techniques are highly beneficial for building more complex GenAI applications, such as RAG- or LLM-powered summarization or data augmentation.

## What will we do in this tutorial?

### Store embeddings and prepare the index

1. Use the [Cohere](https://cohere.com/) Embed Jobs to generate vector embeddings for the first time on large datasets in an asynchronous and scheduled manner.
2. Add vector embeddings into MongoDB Atlas, which can store and index these vector embeddings alongside your other operational/metadata. 
3. Finally, prepare the indexes for both vector embeddings and full-text search on our private dataset.

### Search with vector embeddings

1. Write a simple Python function to accept search terms/phrases and pass it through the Cohere embed API again to get a query vector.
2. Take these resultant query vector embeddings and perform a vector search query using the [$vectorsearch](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/) operator in the MongoDB Aggregation Pipeline.
3. Pre-filter documents using meta information to narrow the search across your dataset, thereby speeding up the performance of vector search results while retaining accuracy.
4. The retrieved semantically similar documents can be post-filtered (relevancy score) to demonstrate a higher degree of control over the semantic search behaviour.

### Search with text and Rerank with Cohere

1. Write a simple Python function to accept search terms/phrases and prepare a query using the $search operator and MongoDB Aggregation Pipeline.
2. Take these resultant documents and perform a reranking operation of the retrieved documents to achieve higher accuracy with full-text search results using the Cohere rerank module.

![Cohere and MongoDB Flow Diagram][1]

This will be a hands-on tutorial that will introduce you to how you can set up MongoDB with **sample_movies** dataset (the link to the file is in the code snippets). You’ll learn how to use the Cohere embedding jobs API to schedule a job to process all the documents as a batch job and update the dataset to add a new field by the name **embedding** that is stored alongside the other metadata/operational data. We will use this field to create a vector search index programmatically using the MongoDB Python drivers. Once we have created this index, we can then demonstrate how to query using the vector embedding as well as perform full-text search using the expressive and composable MongoDB Aggregation Pipeline (Query API). 

## Steps to initialize and run through the tutorial

### Python dependencies

- `pandas`: Helps with data preprocessing and handling
- `cohere`: For embedding model and rerank module
- `pymongo`: For the MongoDB Atlas vector store and full text search
- `s3fs` : To load files directly from s3 bucket

### Install all dependencies

The following line of code is to be run on Jupyter Notebook to install the required packages.

```
!pip install cohere==4.57 pymongo pandas s3fs
```

### Initialize the Cohere API key and MongoDB connection string

If you have not created an API key on the Cohere platform, you can sign up for a Cohere account and create an API key, which you can generate from one of the following interfaces:

- Cohere [dashboard](https://os.cohere.ai/register?ref=txt.cohere.com&__hstc=14363112.fb39cf5aec47995e64cd26603e2e04d9.1682489949734.1683512904818.1683517385804.31&__hssc=14363112.54.1683517385804&__hsfp=3640182760)
- Cohere [CLI tool](https://docs.cohere.ai/cli-key?ref=txt.cohere.com&__hstc=14363112.fb39cf5aec47995e64cd26603e2e04d9.1682489949734.1683512904818.1683517385804.31&__hssc=14363112.54.1683517385804&__hsfp=3640182760)

Also, if you have not created a MongoDB Atlas instance for yourself, you can follow the [tutorial](https://www.mongodb.com/basics/create-database) to create one. This will provide you with your [`MONGODB_CONNECTION_STR`](https://www.mongodb.com/basics/mongodb-connection-string). 

Run the following lines of code in Jupyter Notebook to initialize the Cohere secret or API key and MongoDB Atlas connection string.

```python
import os
import getpass
# cohere api key
try:
    cohere_api_key = os.environ["COHERE_API_KEY"]
except KeyError:
    cohere_api_key = getpass.getpass("Please enter your COHERE API KEY (hit enter): ")

# MongoDB connection string
try:
    MONGO_CONN_STR = os.environ["MONGODB_CONNECTION_STR"]
except KeyError:
    MONGO_CONN = getpass.getpass("Please enter your MongoDB Atlas Connection String (hit enter): ")
```

### Load dataset from the S3 bucket

Run the following lines of code in Jupyter Notebook to read data from an AWS S3 bucket directly to a pandas dataframe.

```python
import pandas as pd
import s3fs
df = pd.read_json("s3://ashwin-partner-bucket/cohere/movies_sample_dataset.jsonl", orient="records", lines=True)
df.to_json("./movies_sample_dataset.jsonl", orient="records", lines=True)
df[:3]
```

![Loaded AWS S3 Dataset][2]

### Initialize and schedule the Cohere embeddings job to embed the "sample_movies" dataset

Here we will create a movies dataset in Cohere by uploading our sample movies dataset that we fetched from the S3 bucket and have stored locally. Once we have created a dataset, we can use the Cohere embed jobs API to schedule a batch job to embed all the entire dataset.

You can run the following lines of code in your Jupyter Notebook to upload your dataset to Cohere and schedule an embedding job.

```python
import cohere  
co_client = cohere.Client(cohere_api_key, client_name='mongodb')
# create a dataset in Cohere Platform
dataset = co_client.create_dataset(name='movies',
                                   data=open("./movies_sample_dataset.jsonl",'r'),
                                   keep_fields=["overview","title","year"],
                                   dataset_type="embed-input").wait()
dataset.wait()
dataset

dataset.wait()
# Schedule an Embedding job to run on the entire movies dataset
embed_job = co_client.create_embed_job(dataset_id=dataset.id, 
    input_type='search_document',
    model='embed-english-v3.0', 
    truncate='END')
embed_job.wait()
output_dataset = co_client.get_dataset(embed_job.output.id)
results = list(map(lambda x:{"text":x["text"], "embedding": x["embeddings"]["float"]},output_dataset))
len(results)
```

### How to initialize MongoDB Atlas and insert data to a MongoDB collection

Now that we have created the vector embeddings for our sample movies dataset, we can initialize the MongoDB client and insert the documents into our collection of choice by running the following lines of code in the Jupyter Notebook.

```python
from pymongo import MongoClient
mongo_client = MongoClient(MONGO_CONN_STR)
# Upload documents along with vector embeddings to MongoDB Atlas Collection
output_collection = mongo_client["sample_mflix"]["cohere_embed_movies"]
if output_collection.count_documents({})>0:
    output_collection.delete_many({})
e = output_collection.insert_many(results)
```

### Programmatically create vector search and full-text search index

With the latest update to the **Pymongo** Python package, you can now create your vector search index as well as full-text search indexes from the Python client itself. You can also create vector indexes using the MongoDB Atlas UI or `mongosh`.

Run the following lines of code in your Jupyter Notebook to create search and vector search indexes on your new collection.

```
output_collection.create_search_index({"definition":
        {"mappings":
         {"dynamic": true,
          "fields": {
            "embedding" : {
                "dimensions": 1024,
                "similarity": "cosine",
                "type": "vector"
                },
            "fullplot":
            }}},
     "name": "default"
    }
)
```

### Query MongoDB vector index using $vectorSearch

MongoDB Atlas brings the flexibility of using vector search alongside full-text search filters. Additionally, you can apply range, string, and numeric filters using the aggregation pipeline. This allows the end user to control the behavior of the semantic search response from the search engine. The below lines of code will demonstrate how you can perform vector search along with pre-filtering on the **year** field to get movies earlier than **1990.** Plus, you have better control over the relevance of returned results, so you can perform post-filtering on the response using the MongoDB Query API. In this demo, we are filtering on the **score** field generated as a result of performing the vector similarity between the query and respective documents, using a heuristic to retain only the accurate results.

Run the below lines of code in Jupyter Notebook to initialize a function that can help you achieve **vector search + pre-filter + post-filter**.

```python
def query_vector_search(q, prefilter = {}, postfilter = {},path="embedding",topK=2):
    ele = co_client.embed(model="embed-english-v3.0",input_type="search_query",texts=[q])
    query_embedding = ele.embeddings[0]
    vs_query = {
                "index": "default",
                "path": path,
                "queryVector": query_embedding,
                "numCandidates": 10,
                "limit": topK,
            }
    if len(prefilter)>0:
        vs_query["filter"] = prefilter
    new_search_query = {"$vectorSearch": vs_query}
    project = {"$project": {"score": {"$meta": "vectorSearchScore"},"_id": 0,"title": 1, "release_date": 1, "overview": 1,"year": 1}}
    if len(postfilter.keys())>0:
        postFilter = {"$match":postfilter}
        res = list(output_collection.aggregate([new_search_query, project, postFilter]))
    else:
        res = list(output_collection.aggregate([new_search_query, project]))
    return res
```

#### Vector search query example

Run the below lines of code in Jupyter Notebook cell and you can see the following results.

```python
query_vector_search("romantic comedy movies", topK=5)
```

![Vector Search Query Example Results][3]

#### Vector search query example with prefilter

```python
query_vector_search("romantic comedy movies", prefilter={"year":{"$lt": 1990}}, topK=5)
```

![Vector Search with Prefilter Example Results][4]

#### Vector search query example with prefilter and postfilter to control the semantic search relevance and behaviour

```python
query_vector_search("romantic comedy movies", prefilter={"year":{"$lt": 1990}}, postfilter={"score": {"$gt":0.76}},topK=5)
```

![Vector Search with Prefilter and Postfilter Example Results][5]

### Leverage MongoDB Atlas full-text search with Cohere Rerank module

[Cohere Rerank](https://txt.cohere.com/rerank/) is a module in the Cohere suite of offerings that enhances the quality of search results by leveraging semantic search. This helps elevate the traditional search engine performance, which relies solely on keywords. Rerank goes a step further by ranking results retrieved from the search engine based on their semantic relevance to the input query. This pass of re-ranking search results helps achieve more appropriate and contextually similar search results.

To demonstrate how the Rerank module can be leveraged with MongoDB Atlas full-text search, we can follow along by running the following line of code in your Jupyter Notebook.

```python
# sample search query using $search operator in aggregation pipeline
def query_fulltext_search(q,topK=25):
    v = {"$search": {
      "text": {
        "query": q,
        "path":"overview"
      }
    }}
    project = {"$project": {"score": {"$meta": "searchScore"},"_id": 0,"title": 1, "release-date": 1, "overview": 1}}
    docs = list(output_collection.aggregate([v,project, {"$limit":topK}]))
    return docs
# results before re ranking
docs = query_fulltext_search("romantic comedy movies", topK=10)
docs
```

![Cohere Rerank Model Sample Results][6]

```python
# After passing the search results through the Cohere rerank module
q = "romantic comedy movies"
docs = query_fulltext_search(q)
results = co_client.rerank(query=q, documents=list(map(lambda x:x["overview"], docs)), top_n=5, model='rerank-english-v2.0') # Change top_n to change the number of results returned. If top_n is not passed, all results will be returned.
for idx, r in enumerate(results):
    print(f"Document Rank: {idx + 1}, Document Index: {r.index}")
    print(f"Document Title: {docs[r.index]['title']}")
    print(f"Document: {r.document['text']}")
    print(f"Relevance Score: {r.relevance_score:.2f}")
    print("\n")
```

Output post reranking the full-text search results:

```
Document Rank: 1, Document Index: 22
Document Title: Love Finds Andy Hardy
Document: A 1938 romantic comedy film which tells the story of a teenage boy who becomes entangled with three different girls all at the same time.
Relevance Score: 0.99


Document Rank: 2, Document Index: 12
Document Title: Seventh Heaven
Document: Seventh Heaven or De zevende zemel is a 1993 Dutch romantic comedy film directed by Jean-Paul Lilienfeld.
Relevance Score: 0.99


Document Rank: 3, Document Index: 19
Document Title: Shared Rooms
Document: A new romantic comedy feature film that brings together three interrelated tales of gay men seeking family, love and sex during the holiday season.
Relevance Score: 0.97


Document Rank: 4, Document Index: 3
Document Title: Too Many Husbands
Document: Romantic comedy adapted from a Somerset Maugham play.
Relevance Score: 0.97


Document Rank: 5, Document Index: 20
Document Title: Walking the Streets of Moscow
Document: "I Am Walking Along Moscow" aka "Ya Shagayu Po Moskve" (1963) is a charming lyrical comedy directed by Georgi Daneliya in 1963 that was nominated for Golden Palm at Cannes Film Festival. Daneliya proved that it is possible to create a masterpiece in the most difficult genre of romantic comedy. Made by the team of young and incredibly talented artists that besides Daneliya included writer/poet Gennady Shpalikov, composer Andrei Petrov, and cinematographer Vadim Yusov (who had made four films with Andrei Tarkovski), and the dream cast of the talented actors even in the smaller cameos, "I Am Walking Along Moscow" keeps walking victoriously through the decades remaining deservingly one of the best and most beloved Russian comedies and simply one of the best Russian movies ever made. Funny and gentle, dreamy and humorous, romantic and realistic, the film is blessed with the eternal youth and will always take to the walk on the streets of Moscow new generations of the grateful viewers.
Relevance Score: 0.96
```

## Summary

In this tutorial, we were able to demonstrate the following:

1. Using the Cohere embedding along with MongoDB Vector Search, we were able to show how easy it is to achieve semantic search functionality alongside your operational data functions.
2. With Cohere Rerank, we were able to search results using full-text search capabilities in MongoDB and then rank them by semantic relevance, thereby delivering richer, more relevant results without replacing your existing search architecture setup.
3. The implementations were achieved with minimal lines of code and showcasing ease of use.
4. Leveraging Cohere Embeddings and Rerank does not need a team of ML experts to develop and maintain. So the monthly costs of maintenance were kept to a minimum.
5. Both solutions are cloud-agnostic and, hence, can be set up on any cloud platform.

The same can be found on a [notebook](https://github.com/prakul/MongoDB-AI-Resources/blob/main/Cohere-MongoDB-Atlas-Vector-Search.ipynb) which will help reduce the time and effort following the steps in this blog.

## What's next?

To learn more about how MongoDB Atlas is helping build application-side ML integration in real-world applications, you can visit the [MongoDB for AI page](https://www.mongodb.com/use-cases/artificial-intelligence).


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blte8f8f2d8681106dd/660c5dfcdd5b9e752ba8949a/1.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt11b31c83a7a30a85/660c5e236c4a398354e46705/2.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf09db7ce89c89f05/660c5e4a3110d0a96d069608/3.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5707998b8d57764c/660c5e75c3bc8bfdfbdd1fc1/4.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt533d00bfde1ec48f/660c5e94c3bc8b26dedd1fcd/5.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc67a9ac477d5029e/660c5eb0df43aaed1cf11e70/6.png

To avoid issues in leveraging AI search functionality or machine learning in your application, Cohere and MongoDB offer fully managed solutions.

How to Use Cohere Embeddings and Rerank Modules With MongoDB Atlas

One of the challenges of e-commerce applications is to provide relevant and personalized product recommendations to customers. Traditional keyword-based search methods often fail to capture the semantic meaning and intent of the user search queries, and return results that do not meet the user’s needs. In turn, they fail to convert into a successful sale. To address this problem, RAG (retrieval-augmented generation) is used as a framework powered by MongoDB Atlas Vector Search, [LangChain](https://www.langchain.com/), and [IBM watsonx.ai](https://www.ibm.com/products/watsonx-ai/foundation-models). 

RAG is a natural language generation (NLG) technique that leverages a retriever module to fetch relevant documents from a large corpus and a generator module to produce text conditioned on the retrieved documents. Here, the RAG framework is used to power product recommendations as an extension to existing semantic search techniques.

- RAG use cases can be easily built using the vector search capabilities of MongoDB Atlas to store and query large-scale product embeddings that represent the features and attributes of each product. Because of MongoDB’s flexible schema, these are stored right alongside the product embeddings, eliminating the complexity and latency of having to retrieve the data from separate tables or databases.
- RAG then retrieves the most similar products to the user query based on the cosine similarity of their embeddings, and generates natural language reasons that highlight why these products are relevant and appealing to the user. 
- RAG can also enhance the user experience (UX) by handling complex and diverse search queries, such as "a cozy sweater for winter" or "a gift for my daughter who is interested in science", and provides accurate and engaging product recommendations that increase customer satisfaction and loyalty.

![MongoDB Vector Search with IBM watsonx][2]

## Why MongoDB Atlas Vector Search?

To enable RAG to retrieve relevant documents, we need a vector search engine that can efficiently store and query large-scale document embeddings. MongoDB Atlas allows us to index documents based on their embeddings and perform similarity search using cosine distance, euclidean distance, or dot product. MongoDB Atlas also provides flexible schema, scalability, security, and performance features that over 45,000 organizations — from startups to enterprises and governments — rely on.

By using RAG with MongoDB Atlas Vector Search, we can enhance the user experience of product recommendations in several ways. First, we can provide more personalized and diverse recommendations that match the user's query or context. Second, we can generate more informative and engaging responses that can explain why each product is recommended and how it compares to others. Third, we can improve the relevance and accuracy of the recommendations by updating the document embeddings as MongoDB is primarily an operational data layer (ODL) that provides transaction features.

## What is watsonx.ai?

[Watsonx.ai](https://www.ibm.com/products/watsonx-ai) is IBM’s next-generation enterprise studio for AI builders, bringing together new generative AI capabilities with traditional machine learning (ML) that span the entire AI lifecycle. With watsonx.ai, you can train, validate, tune, and deploy foundation and traditional ML models.

watsonx.ai brings forth a [curated library of foundation models](https://www.ibm.com/products/watsonx-ai/foundation-models), including IBM-developed models, open-source models, and models sourced from third-party providers. Not all models are created equal, and the curated library provides enterprises with the optionality to select the model best suited to a particular use case, industry, domain, or even price performance. Further, IBM-developed models, such as the [Granite model series](https://newsroom.ibm.com/2023-09-28-IBM-Announces-Availability-of-watsonx-Granite-Model-Series,-Client-Protections-for-IBM-watsonx-Models), offer another level of enterprise-readiness, transparency, and indemnification for production use cases. We’ll be using Granite models in our demonstration. For the interested reader, IBM has [published information about](https://www.ibm.com/downloads/cas/X9W4O6BM) its data and training methodology for its Granite foundation models.

## How to build a custom RAG-powered product discovery pipeline

For this tutorial, we will be using an [e-commerce products dataset](https://raw.githubusercontent.com/ashwin-gangadhar-mdb/mdb-watsonx-rag/320a21a951875709febbb48d9fbcbee7eff167aa/amazon-products.jsonl) containing over 10,000 product details. We will be using the [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) model from Hugging Face to generate the vector embeddings to store and retrieve product information. You will need a Python notebook or an IDE, a MongoDB Atlas account, and a [wastonx.ai](https://www.ibm.com/products/watsonx-ai) account for hands-on experience.

For convenience, the notebook to follow along and execute in your environment is available on [GitHub](https://github.com/ashwin-gangadhar-mdb/mdb-watsonx-rag).

### Python dependencies

* `langchain`: Orchestration framework

* `ibm-watson-machine-learning`: For IBM LLMs

* `wget`: To download knowledge base data

* `sentence-transformers`: For embedding model

* `pymongo`: For the MongoDB Atlas vector store

### watsonx.ai dependencies

We’ll be using the watsonx.ai foundation models and Python SDK to implement our RAG pipeline in LangChain.

1. **Sign up for a free watsonx.ai trial on IBM cloud**. Register and [get set up](https://dataplatform.cloud.ibm.com/registration/stepone?context=wx).
2. **Create a watsonx.ai Project**. During onboarding, a sandbox project can be quickly created for you. You can either use the sandbox project or [create one](https://dataplatform.cloud.ibm.com/projects/?context=wx); the link will work once you have registered and set up watsonx.ai. If more help is needed, you can read the [documentation](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/projects.html?context=wx&audience=wdp).
3. **Create an API key to access watsonx.ai foundation models**. Follow [the steps](https://cloud.ibm.com/docs/account?topic=account-userapikey&interface=ui) to create your API key.
4. **Install and use watsonx.ai**. Also known as the IBM Watson Machine Learning SDK, [watsonx.ai SDK information](https://ibm.github.io/watson-machine-learning-sdk/foundation_models.html) is available on [GitHub](https://ibm.github.io/watson-machine-learning-sdk/foundation_models.html). Like any other Python module, you can install it with a pip install. Our example notebook takes care of this for you.

We will be running all the code snippets below in a Jupyter notebook. You can choose to run these on VS Code or any other IDE of your choice.

**Initialize the LLM**

Initialize the watsonx URL to connect by running the below code blocks in your Jupyter notebook:

```python
# watsonx URL

try:
    wxa_url = os.environ["WXA_URL"]

except KeyError:
    wxa_url = getpass.getpass("Please enter your watsonx.ai URL domain (hit enter): ")
```

Enter the URL for accessing the watsonx URL domain. For example: https://us-south.ml.cloud.ibm.com.

To be able to access the LLM models and other AI services on watsonx, you need to initialize the API key. You init the API key by running the following code block in you Jupyter notebook:

```python
# watsonx API Key

try:
    wxa_api_key = os.environ["WXA_API_KEY"]
except KeyError:
    wxa_api_key = getpass.getpass("Please enter your watsonx.ai API key (hit enter): ")
```

You will be prompted when you run the above code to add the IAM API key you fetched earlier.

Each experiment can tagged or executed under specific projects. To fetch the relevant project, we can initialize the project ID by running the below code block in the Jupyter notebook:

```python
# watsonx Project ID

try:
    wxa_project_id = os.environ["WXA_PROJECT_ID"]
except KeyError:
    wxa_project_id = getpass.getpass("Please enter your watsonx.ai Project ID (hit enter): ")
```

You can find the project ID alongside your IAM API key in the settings panel in the watsonx.ai portal.

**Language model**

In the code example below, we will initialize Granite LLM from IBM and then demonstrate how to use the initialized LLM with the LangChain framework before we build our RAG.

We will use the query: "I want to introduce my daughter to science and spark her enthusiasm. What kind of gifts should I get her?" 

This will help us demonstrate how the LLM and vector search work in an RAG framework at each step.

Firstly, let us initialize the LLM hosted on the watsonx cloud. To access the relevant Granite model from watsonx, you need to run the following code block to initialize and test the model with our sample query in the Jupyter notebook: 

```python
from ibm_watson_machine_learning.foundation_models.utils.enums import ModelTypes
from ibm_watson_machine_learning.foundation_models import Model
from ibm_watson_machine_learning.metanames import GenTextParamsMetaNames as GenParams
from ibm_watson_machine_learning.foundation_models.utils.enums import DecodingMethods

parameters = {
    GenParams.DECODING_METHOD: DecodingMethods.GREEDY,
    GenParams.MIN_NEW_TOKENS: 1,
    GenParams.MAX_NEW_TOKENS: 100
}

model = Model(
    model_id=ModelTypes.GRANITE_13B_INSTRUCT,
    params=parameters,
    credentials={
        "url": wxa_url,
        "apikey": wxa_api_key
    },
    project_id=wxa_project_id
)

from ibm_watson_machine_learning.foundation_models.extensions.langchain import WatsonxLLM

granite_llm_ibm = WatsonxLLM(model=model)

# Sample query chosen in the example to evaluate the RAG use case
query = "I want to introduce my daughter to science and spark her enthusiasm. What kind of gifts should I get her?"

# Sample LLM query without RAG framework
result = granite_llm_ibm(query)
```

Output:

![Jupyter Notebook Output][3]

### Initialize MongoDB Atlas for vector search

Prior to starting this section, you should have already set up a cluster in MongoDB Atlas. If you have not created one for yourself, then you can follow the steps in the [MongoDB Atlas tutorial](https://www.mongodb.com/basics/mongodb-atlas-tutorial) to create an account in Atlas and a cluster with which we can store and retrieve data. It is also advised that the users spin an Atlas dedicated cluster with size M10 or higher for this tutorial.

Now, let us see how we can set up MongoDB Atlas to provide relevant information to augment our RAG framework. 

**Init Mongo client**

We can connect to the MongoDB Atlas cluster using the connection string as detailed in the tutorial link above. To initialize the connection string, run the below code block in your Jupyter notebook:

```python
from pymongo import MongoClient

try:
    MONGO_CONN = os.environ["MONGO_CONN"]
except KeyError:
    MONGO_CONN = getpass.getpass("Please enter your MongoDB connection String (hit enter): ")
```

When prompted, you can enter your MongoDB Atlas connection string.

**Download and load data to MongoDB Atlas**

In the steps below, we demonstrate how to download the products dataset from the provided URL link and add the documents to the respective collection in MongoDB Atlas. We will also be embedding the raw product texts as vectors before adding them in MongoDB. You can do this by running the following lines of code your Jupyter notebook:

```python
import wget

filename = './amazon-products.jsonl'
url = "https://github.com/ashwin-gangadhar-mdb/mbd-watson-rag/raw/main/amazon-products.jsonl"

if not os.path.isfile(filename):
    wget.download(url, out=filename)

# Load the documents using Langchain Document Loader
from langchain.document_loaders import JSONLoader

loader = JSONLoader(file_path=filename, jq_schema=".text",text_content=False,json_lines=True)
docs = loader.load()

# Initialize Embedding for transforming raw documents to vectors**
from langchain.embeddings import HuggingFaceEmbeddings
from tqdm import tqdm as notebook_tqdm

embeddings = HuggingFaceEmbeddings()

# Initialize MongoDB client along with Langchain connector module
from langchain.vectorstores import MongoDBAtlasVectorSearch

client = MongoClient(MONGO_CONN)
vcol = client["amazon"]["products"]
vectorstore = MongoDBAtlasVectorSearch(vcol, embeddings)

# Load documents to collection in MongoDB**
vectorstore.add_documents(docs)
```

You will be able to see the documents have been created in `amazon` database under the collection `products`.

![MongoDB Atlas Products Collection][4]

Now all the product information is added to the respective collection, we can go ahead and create a vector index by following the steps given in the [Atlas Search index tutorial](https://www.mongodb.com/docs/atlas/atlas-search/create-index/). You can create the search index using both the Atlas UI as well as programmatically. Let us look at the steps if we are doing this using the Atlas UI.

![Create Atlas Search Index][5]

You can select the Json Editor view to insert the vector search index config as shown in the image. Insert the following mapping to create the vector indexes.

```json
{ 
    "mappings": {   
        "dynamic": true,   
        "fields": {     
            "embedding": [       
                {         
                    "dimensions": 768,         
                    "similarity": "cosine",         
                    "type": "knnVector"       
                }      
            ]   
        } 
    } 
} 
```

![Atlas Search Index JSON][6] 

Once you have updated the vector search indexes in the Atlas UI, this will start an automatic process to create the vector search indexes for the collection for the first time. Also, this will automatically index any new document added to the collection. Seamlessly, you will be able to update the collection in real time as you add new products and augment the RAG framework so that users are always served with the latest and greatest products you are offering.

For a description of the other fields in this configuration, you can check out our [Vector Search documentation](https://www.mongodb.com/docs/atlas/atlas-search/vector-search/).

**Sample query to vector search**

We can test the vector similarity search by running the sample query with the LangChain MongoDB Atlas Vector Search connector. Run the following code in your Jupyter notebook:

```python
texts_sim = vectorstore.similarity_search(query, k=3)

print("Number of relevant texts: " + str(len(texts_sim)))
print("First 100 characters of relevant texts.")

for i in range(len(texts_sim)):
    print("Text " + str(i) + ": " + str(texts_sim[i].page_content[0:100]))
```

![Sample Vector Search Query Output][7]

In the above example code, we are able to use our sample text query to retrieve three relevant products. Further in the tutorial, let’s see how we can combine the capabilities of LLMs and vector search to build a RAG framework. For further information on various operations you can perform with the `MongoDBAtlasVectorSearch` module in LangChain, you can visit the [Atlas Vector Search documentation](https://api.python.langchain.com/en/latest/vectorstores/langchain.vectorstores.mongodb_atlas.MongoDBAtlasVectorSearch.html).

### RAG chain

In the code snippets below, we demonstrate how to initialize and query the RAG chain. We also introduce methods to improve the output from RAG so you can customize your output to cater to specific needs, such as the reason behind the product recommendation, language translation, summarization, etc.

So, you can set up the RAG chain and execute to get the response for our sample query by running the following lines of code in your Jupyter notebook:

```python
from langchain.chains import RetrievalQA
from langchain.prompts import PromptTemplate

prompt = PromptTemplate(template="""

Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

{context}

##Question:{question} \n\

##Top 3 recommnedations from Products and reason:\n""",input_variables=["context","question"])

chain_type_kwargs = {"prompt": prompt}
retriever = vectorstore.as_retriever(search_type="mmr", search_kwargs={'k': 6, 'lambda_mult': 0.25})

qa = RetrievalQA.from_chain_type(llm=granite_llm_ibm, chain_type="stuff",
                                 retriever=retriever,
                                 chain_type_kwargs=chain_type_kwargs)

res = qa.run(query)

print(f"{'-'*50}")
print("Query: " + query)
print(f"Response:\n{res}\n{'-'*50}\n")
```
The output will look like this:

![Sample RAG Chain Output][8]

You can see from the example output where the RAG is able to recommend products based on the query as well as provide a reasoning or explanation as to how this product suggestion is relevant to the query, thereby enhancing the user experience.

## Conclusion

In this tutorial, we demonstrated how to use watsonx LLMs along with Atlas Vector Search to build a RAG framework. We also demonstrated how to efficiently use the RAG framework to customize your application needs, such as the reasoning for product suggestions. By following the steps in the article, we were also able to bring the power of machine learning models to a private knowledge base that is stored in the Atlas data platform.

In summary, RAG is a powerful NLG technique that can generate product recommendations as an extension to semantic search using vector search capabilities provided by MongoDB Atlas. RAG can also improve the UX of product recommendations by providing more personalized, diverse, informative, and engaging descriptions.

## Next steps

Explore more details on how you can [build generative AI applications](https://www.mongodb.com/developer/products/atlas/building-generative-ai-applications-vector-search-open-source-models/) using various assisted technologies and MongoDB Atlas Vector Search.

To learn more about Atlas Vector Search, visit the [product page](https://www.mongodb.com/products/platform/atlas-vector-search) or the documentation for [creating a vector search index](https://www.mongodb.com/docs/atlas/atlas-search/field-types/knn-vector/) or [running vector search queries](https://www.mongodb.com/docs/atlas/atlas-search/knn-beta/).

To learn more about watsonx, visit the [IBM watsonx page](https://www.ibm.com/watsonx).


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltabec5b11a292b3d6/6553a958c787a446c12ab071/image1.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blte019110c36fc59f5/6553a916c787a4d4282ab069/image3.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltabec5b11a292b3d6/6553a958c787a446c12ab071/image1.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blta69be6d193654a53/6553a9ad4d2859f3c8afae47/image5.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt70003655ac1919b7/6553a9d99f2b9963f6bc99de/image6.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltcbdae931b43cc17a/6553a9f788cbda51858566f6/image2.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt0b43cc03bf7bb27f/6553aa124452cc3ed9f9523d/image7.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltf6c21ef667b8470b/6553aa339f2b993db7bc99e3/image4.png

Learn how to build a RAG framework using MongoDB Atlas Vector Search and IBM watsonx LLMs.

How to Seamlessly Use MongoDB Atlas and IBM watsonx.ai LLMs in Your GenAI Applications

Vittal Pai

Senior Solutions Architect

Francesco Baldissera

Industry Solutions Consultant

In the vast realm of Ecommerce, customers' ability to quickly and accurately search through an extensive range of products is paramount. [Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) is emerging as a turning point in this space, offering a refined approach to search that goes beyond mere keyword matching. Let's delve into its implementation using [MongoDB Atlas](https://www.mongodb.com/atlas/database), Atlas Vector Search, and [Databricks](https://www.databricks.com/).

### Prerequisites

* [MongoDB Atlas cluster](https://www.mongodb.com/docs/atlas/getting-started/)
* [Databricks cluster](https://docs.databricks.com/en/clusters/index.html)
* [python>=3.7](https://www.python.org/downloads/release/python-370/)
* [pip3](https://pypi.org/project/pip/)
* [Node.js and npm](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm)
* [GitHub repo](https://github.com/mongodb-industry-solutions/retail-pricing-search-personaliztion) for AI-enhanced search and vector search (code is bundled up for clarity)

In a previous tutorial, [Learn to Build AI-Enhanced Retail Search Solutions with MongoDB and Databricks](https://www.mongodb.com/developer/products/atlas/retail-search-mongodb-databricks/), we showcased how the integration of MongoDB and Databricks provides a comprehensive solution for the retail industry by combining real-time data processing, workflow orchestration, machine learning, custom data functions, and advanced search capabilities as a way to optimize product catalog management and enhance customer interactions.

In this tutorial, we are going to be building the Vector Search solution on top of the codebase from the previous tutorial. Please check out the Github repository for the [full solution](https://github.com/mongodb-industry-solutions/retail-pricing-search-personaliztion/tree/main).

The diagram below represents the Databricks workflow for indexing data from the atp (available to promise), images, prd_desc (product discount), prd_score (product score), and price collections. These collections are also part of the previously mentioned tutorial, so please refer back if you need to access them.

Within the MongoDB Atlas platform, we can use change streams and the [MongoDB Connector for Spark](https://www.mongodb.com/docs/spark-connector/current/) to move data from the collections into a new collection called Catalog. From there, we will use a text transformer to create the **`Catalog Final Collection`**. This will enable us to create a corpus of indexed and vector embedded data that will be used later as the search dictionary. We’ll call this collection **`catalog_final_myn`**. This will be shown further along after we embed the product names.

The catalog final collection will include the available to promise status for each product, its images, the product discount, product relevance score, and price, along with the vectorized or embedded product name that we’ll point our vector search engine at.

With the image below, we explain what the [Databricks workflow](https://docs.databricks.com/en/workflows/index.html) looks like. It consists of two jobs that are separated in two notebooks respectively. We’ll go over each of the notebooks below.

![Overview of a Databricks job, including two pipelines to ingest data from MongoDB collections, transform that data, and vectorize it using a text transformer model](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc3b2a88fc9350ffd/650db4f5c9f9dc0b117b978a/vectorsearch1.png)

## Indexing and merging several collections into one catalog

The first step is to ingest data from the previously mentioned collections using the spark.readStream method from the MongoDB Connector for Spark. The code below is part of the notebook we’ll set as a job using Databricks Workflows. You can learn more about [Databricks notebooks](https://docs.databricks.com/en/notebooks/index.html) by following their tutorial. 

```
atp = spark.readStream.format("mongodb").\ option('spark.mongodb.connection.uri', MONGO_CONN).\ option('spark.mongodb.database', "search").\ option('spark.mongodb.collection', "atp_status_myn").\ option('spark.mongodb.change.stream.publish.full.document.only','true').\ option('spark.mongodb.aggregation.pipeline',[]).\ option("forceDeleteTempCheckpointLocation", "true").load() atp = atp.drop("_id") atp.writeStream.format("mongodb").\ option('spark.mongodb.connection.uri', MONGO_CONN).\ option('spark.mongodb.database', "search").\ option('spark.mongodb.collection', "catalog_myn").\ option('spark.mongodb.operationType', "update").\ option('spark.mongodb.upsertDocument', True).\ option('spark.mongodb.idFieldList', "id").\ option("forceDeleteTempCheckpointLocation", "true").\ option("checkpointLocation", "/tmp/retail-atp-myn4/_checkpoint/").\ outputMode("append").\ start()
```
This part of the notebook reads data changes from the atp_status_myn collection in the search database, drops the _id field, and then writes (or updates) the processed data to the catalog_myn collection in the same database. 

Notice how it’s reading from the `atp_status_myn` collection, which already has the one hot encoding (boolean values if the product is available or not) from the [previous  tutorial](https://docs.google.com/document/d/1KbnO00DJrlZIXdWS_QCxfHQsbDB9jMm1z_5q44JYx78/edit?usp=sharing). This way, we make sure that we only embed the data from the products that are available in our stock.

Please refer to [the full notebook in our Github repository](https://github.com/mongodb-industry-solutions/retail-pricing-search-personaliztion/blob/main/notebooks/catalogIndexing.ipynb) if you want to learn more about all the data ingestion and transformations conducted during this stage. 

## Encoding text as vectors and building the final catalog collection

Using a combination of Python libraries and PySpark operations to process data from the Catalog MongoDB collection, we’ll transform it, vectorize it, and write the transformed data back to the Final Catalog collection. On top of this, we’ll build our application search business logic.

We start by using the %pip magic command, which is specific to Jupyter notebooks and IPython environments. The necessary packages are:
* **pymongo:** A Python driver for MongoDB.
* **tqdm:** A library to display progress bars.
* **sentence-transformers:** A library for state-of-the-art sentence, text, and image embeddings.

First, let’s use pip to install these packages in our Databricks notebook:

```
%pip install pymongo tqdm sentence-transformers
```

We continue the notebook with the following code: 

```
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
```

Here we load a [pre-trained model](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) from the sentence-transformers library. This model will be used to convert text into embeddings or vectors. 

The next step is to bring the data from the MongoDB Atlas catalog and search collections. This as a continuation of the same notebook:
```
catalog_status = spark.readStream.format("mongodb").\
            option('spark.mongodb.connection.uri', MONGO_CONN).\
            option('spark.mongodb.database', "search").\
            option('spark.mongodb.collection', "catalog_myn").\
option('spark.mongodb.change.stream.publish.full.document.only','true').\
            option('spark.mongodb.aggregation.pipeline',[]).\
            option("forceDeleteTempCheckpointLocation", "true").load()
```
With this code, we set up a structured streaming read from the **`catalog_myn`** collection in the **`search`** database of MongoDB. The resulting data is stored in the **`catalog_status`** DataFrame in Spark. The read operation is configured to fetch the full document from MongoDB's change stream and does not apply any aggregation.

The notebook code continues with: 
```
#Calculating new column called discounted price using the F decorator

catalog_status = catalog_status.withColumn("discountedPrice", F.col("price") * F.col("pred_price"))

#One hot encoding of the atp_status column 

catalog_status = catalog_status.withColumn("atp", (F.col("atp").cast("boolean") & F.lit(1).cast("boolean")).cast("integer"))

#Running embeddings of the product titles with the get_vec function

catalog_status.withColumn("vec", get_vec("title"))

#Dropping _id column and creating a new final catalog collection with checkpointing

catalog_status = catalog_status.drop("_id")
catalog_status.writeStream.format("mongodb").\
            option('spark.mongodb.connection.uri', MONGO_CONN).\
            option('spark.mongodb.database', "search").\
            option('spark.mongodb.collection', "catalog_final_myn").\
            option('spark.mongodb.operationType', "update").\
            option('spark.mongodb.idFieldList', "id").\
            option("forceDeleteTempCheckpointLocation", "true").\
            option("checkpointLocation", "/tmp/retail-atp-myn5/_checkpoint/").\
            outputMode("append").\
            start()
```

With this last part of the code, we calculate a new column called discountedPrice as the product of the predicted price. Then, we perform [one-hot encoding](https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/) on the atp status column, vectorize the title of the product, and merge everything back into a final catalog collection.

Now that we have our catalog collection with its proper embeddings, it’s time for us to build the Vector Search Index using [MongoDB Atlas Search](https://www.mongodb.com/docs/atlas/atlas-search/). 

## Configuring the Atlas Vector Search index 

Here we’ll define how data should be stored and indexed for efficient searching. To configure the index, you can insert the snippet in MongoDB Atlas by browsing to your cluster splash page and clicking on the “Search” tab:
![Overview of the Search configuration panel for MongoDB Atlas collections](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt899d2c0d7b2a4268/650db8d9a0bdc093216ef75b/vectorsearch2.png)

Next, you can click over “Create Index.” Make sure you select “JSON Editor”:
![Overview of the JSON Editor functionality in the Search Configuration panel for MongoDB Atlas ](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt761abc22dd17f1fb/650db9239a515222b68bec9c/vectorsearch3.png)

Paste the JSON snippet from below into the JSON Editor. Make sure you select the correct database and collection! In our case, the collection name is **`catalog_final_myn`**. Please refer to the [full code in the repository](https://github.com/mongodb-industry-solutions/retail-pricing-search-personaliztion/blob/main/mdbindex/search.catalog_final_myn) to see how the full index looks and how you can bring it together with the rest of parameters for the [AI-enhanced search tutorial](https://www.mongodb.com/developer/products/atlas/retail-search-mongodb-databricks/).

```
{
  "mappings": {
    "fields": {
            "vec": [
        {
          "dimensions": 384,
          "similarity": "cosine",
          "type": "knnVector"
        }
      ]
    }
  }
}
```

In the code above, the vec field is of type [knnVector](https://www.mongodb.com/docs/atlas/atlas-search/field-types/knn-vector/#std-label-bson-data-types-knn-vector), designed for vector search. It indicates that each vector has 384 dimensions and uses cosine similarity to determine vector closeness. This is crucial for semantic search, where the goal is to find results that are contextually or semantically related.

By implementing these indexing parameters, we speed up retrieval times. Especially with high-dimensional vector data, as raw vectors can consume a significant amount of storage and reduce the computational cost of operations like similarity calculations. 

Instead of comparing a query vector with every vector in the dataset, indexing allows the system to compare with a subset, saving computational resources.

## A quick example of improved search results

[Browse over to our LEAFYY Ecommerce website](http://ec2-54-216-127-120.eu-west-1.compute.amazonaws.com:3000/), in which we will perform a search for the keywords ``tan bags``. You’ll get these results: 

![Search engine results page after running a simple query on an Ecommerce storefront powered with MongoDB Atlas Search](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7a45a53f40d1e81e/650db9f4b258105f06802ddc/vectorsearch4.png)

As you can see, you’ll first get results that match the specific tokenized keywords “tan” and “bags”. As a result, this will give you any product that contains any or both of those keywords in the product catalog collection documents. 

However, not all the results are bags or of tan color. You can see shoes, wallets, a dress, and a pair of pants. This could be frustrating as a customer, prompting them to leave the site.

Now, enable vector search by clicking on the checkbox on the left of the magnifying glass icon in the search bar, and re-run the query “tan bags”. The results you get are in the image below: 
![Search engine results page after running a vector search query on an ecommerce storefront powered with MongoDB Atlas Search](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt832782bc8dc0904a/650dba3427287d7d4dbf78de/vectorsearch5.png)

As you can see from the screenshot, the results became more relevant for a consumer. Our search engine is able to identify similar products by understanding the context that “beige” is a similar color to “tan”, and therefore showcase these products as alternatives.

## Conclusion

By working with MongoDB Atlas and Databricks, we can create real-time data transformation pipelines. We achieve this by leveraging the MongoDB Connector for Spark to prepare our operational data for vectorization, and store it back into our MongoDB Atlas collections. This approach allows us to develop the search logic for our Ecommerce app with minimal operational overhead.

On top of that, Atlas Vector Search provides a robust solution for implementing advanced search features, making it easy to deliver a great search user experience for your customers. By understanding and integrating these tools, developers can create search experiences that are fast, relevant, and user-friendly. 

Make sure to review the [full code in our GitHub repository](https://github.com/mongodb-industry-solutions/retail-pricing-search-personaliztion). [Contact us](https://www.mongodb.com/community/forums/) to get a deeper understanding of how to build advanced search solutions for your Ecommerce business. 


  

Learn how to implement Databricks Workflows and Atlas Vector Search for your Ecommerce accuracy.

How to Implement Databricks Workflows and Atlas Vector Search for Enhanced Ecommerce Search Accuracy

Learn how to utilize MongoDB and Databricks to build ai-enhanced retail search solutions.

Learn to Build AI-Enhanced Retail Search Solutions with MongoDB and Databricks

Venkatesh Shanbhag

Partner Solution Architect

The application development landscape is evolving very rapidly. Today, users crave intuitive, context-aware experiences that understand their intent and deliver relevant results even when queries aren't perfectly phrased, putting an end to the keyword-based search practices. This is where MongoDB Atlas and Google Cloud Vertex AI can help users build and deploy scalable and resilient applications. 

[MongoDB Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) is a cutting-edge tool that indexes and stores high-dimensional vectors, representing the essence of your data. It allows you to perform lightning-fast similarity searches, retrieving results based on meaning and context. Google Vertex AI is a comprehensive AI platform that houses an abundance of pre-trained models and tools, including the powerful [Vertex AI](https://cloud.google.com/vertex-ai/docs) PALM. This language model excels at extracting semantic representations from text data, generating those crucial vectors that fuel MongoDB Atlas Vector Search. 

Vector Search can be useful in a variety of contexts, such as natural language processing and recommendation systems. It is a powerful technique that can be used to find similar data based on its meaning.

In this tutorial, we will see how to get started with MongoDB Atlas and Vertex AI. If you are new to MongoDB Atlas, refer to the [documentation](https://console.cloud.google.com/marketplace/product/mongodb/mdb-atlas-self-service?utm_source=mongodb_devrel&utm_medium=static_site&utm_campaign=devrel&utm_id=mongodb) to get set up from Google Cloud Marketplace or use the [Atlas registration page](https://account.mongodb.com/account/register). 

## Before we begin

Make sure that you have the below prerequisites set up before starting to test your application.

1. MongoDB Atlas access, either by the [registration page](https://account.mongodb.com/account/register) or from [Google Cloud Marketplace](https://console.cloud.google.com/marketplace/product/mongodb/mdb-atlas-self-service?project=mdb-gcp-marketplace&utm_source=nurture+&utm_medium=email+&utm_campaign=Int_EM_GC+drip+series+dev+1_12_23_WW&utm_id=mongodb&elqTrackId=A66D8FD3837BB72F4DBB8CABAC0A9F37&elqTrack=true)
2. Access to [Google Cloud Project](https://console.cloud.google.com/welcome) to deploy and create a [Compute Engine instance](https://console.cloud.google.com/compute/instances)

## How to get set up

Let us consider a use case where we are loading sample PDF documents to MongoDB Atlas as vectors and deploying an application on Google Cloud to perform a vector search on the PDF documents. 

We will start with the creation of MongoDB Atlas Vector Search Index on the collection to store and retrieve the vectors generated by the Google Vertex AI PALM model. To store and access vectors on MongoDB Atlas, we need to create an [Atlas Search index](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-tutorial/?lb-height=100%&lb-width=100%&presentation=true#prerequisites). 

### Create an Atlas Search index 

1. Navigate to the **Database Deployments** page for your project.
2. Click on **Create Database.** Name your Database **vertexaiApp** and your collection **chat-vec**.
3. Click **Atlas Search** from the Services menu in the navigation bar.
4. Click **Create Search Index** and select **JSON Editor** under **Atlas Vector Search**. Then, click **Next.**
5. In the **Database and Collection** section, find the database **vertexaiApp**, and select the **chat-vec** collection.
6. Replace the default definition with the following index definition and then click **Next**. Click on **Create Search index** on the review page.

```json
{
    "fields": [
        {
            "type":"vector",
            "path":"vec",
            "numDimensions":768,
            "similarity": "cosine"
        }
    ]
}
```

### Create a Google Cloud Compute instance

We will create a [Google Cloud virtual machine instance](https://cloud.google.com/compute/docs/instances/create-start-instance#console) to run and deploy the application. The Google Cloud VM can have all the default configurations. To begin, log into your Google Cloud Console and perform the following steps:

- In the Google Cloud console, click on **Navigation menu > Compute Engine.**
- Create a new VM instance with the below configurations:
    - **Name:** vertexai-chatapp
    - **Region**: region near your physical location
    - **Machine configurations:**
        - Machine type: High Memory, n1-standard-1
- Boot disk: Click on **CHANGE**
    - Increase the size to 100 GB.
    - Leave the other options to default (Debian).
- Access: Select **Allow full access** to all Cloud APIs.
- Firewall: Select all.
- Advanced options:
    - Networking: Expand the default network interface.
    - For External IP range: Expand the section and click on **RESERVE STATIC EXTERNAL IP ADDRESS**. This will help users to access the deployed application from the internet.
    - Name your IP and click on **Done**.
- Click on **CREATE** and the VM will be created in about two to three minutes.

### Deploy the application

Once the VM instance is created, SSH into the VM instance and clone the [GitHub repository](https://github.com/mongodb-partners/MongoDB-VertexAI-Qwiklab.git).

```
git clone https://github.com/mongodb-partners/MongoDB-VertexAI-Qwiklab.git
```

The repository contains a script to create and deploy a [Streamlit](https://streamlit.io/) application to transform and store PDFs in MongoDB Atlas, then search them lightning-fast with Atlas Vector Search. The app.py script in the repository uses Python and LangChain to leverage MongoDB Atlas as our data source and Google Vertex AI for generating embeddings. 

We start by setting up connections and then utilize LangChain’s ChatVertexAI and Google's Vertex AI embeddings to transform the PDF being loaded into searchable vectors. Finally, we constructed the Streamlit app structure, enabling users to input queries and view the top retrieved documents based on vector similarity.

Install the required dependencies on your virtual machine using the below commands:

```bash
sudo apt update
sudo apt install python3-pip
sudo apt install git
git --version
pip3 --version
cd MongoDB-VertexAI-Qwiklab
pip3 install -r requirements.txt
```

Once the requirements are installed, you can run the application using the below command. Open the application using the public IP of your VM and the port mentioned in the command output:

```bash
streamlit run app.py
```

![Screenshot of the application for uploading the documents][1]

The application uses the *PdfReader* library to read the PDF file from the user and chunks the data into batches. Each chunk is converted into vectors using LangChain’s *VertexAIEmbeddings* libraries. The uploaded PDF is written into MongoDB Atlas in chunks, along with the vectors. The data can be viewed in vertexaiApp.chat-vec namespace documents.

![Screenshot of the vectors being stored into MongoDB Atlas][2]

To perform a search on the data, navigate to the Q&A tab on your application. Type in your query and press enter. The application will perform a vector search on MongoDB documents using the index created earlier. The application uses the chat-bison model to converse with the user.

![Screenshot of the application for searching the PDF data from MongoDB using Atlas Vector Search][3]

## Conclusion

In this tutorial, you learned how to build a simple application using MongoDB Atlas and Google Cloud Vertex AI using LangChain to access MongoDB Atlas. Also, we have seen how to deploy the application on Google Cloud Compute instance and access it using the public static IP address allocated. 

Try MongoDB Atlas for free in [Google Cloud Marketplace](https://console.cloud.google.com/marketplace/product/mongodb/mdb-atlas-self-service?project=mdb-gcp-marketplace&utm_source=nurture+&utm_medium=email+&utm_campaign=Int_EM_GC+drip+series+dev+1_12_23_WW&utm_id=mongodb&elqTrackId=A66D8FD3837BB72F4DBB8CABAC0A9F37&elqTrack=true) using our pay-as-you-go model and take advantage of our simplified billing.


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blted03916b8de19681/65a7ede75d51352518fb89d6/1.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltacbaf40c14ecccbc/65a7ee01cdbb961f8ac4faa4/2.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc7a3b3c7a2db880c/65a7ee187a1dd7b984e136ba/3.png

Leverage Atlas Vector Search to perform semantic search, Google Vertex AI for AI capabilities, and LangChain for integration to build smart applications.

Build Smart Applications With Atlas Vector Search and Google Vertex AI

Shiv Pullepu

Principal, Financial Industry Solutions

Rajesh Vinayagam

Card fraud is a significant problem and fear for both consumers and businesses.  However, despite the seriousness of it, there are solutions that can be implemented for card fraud prevention. Financial institutions have various processes and technical solutions in place to detect and prevent card fraud, such as monitoring transactions for suspicious activity, implementing know-your-customer (KYC) procedures, and a combination of controls based on static rules or machine learning models. These can all help, but they are not without their own challenges. 

Financial institutions with legacy fraud prevention systems can find themselves fighting against their own data infrastructure. These challenges can include:

* **Incomplete data**: Legacy systems may not have access to all relevant data sources, leading to a lack of visibility into fraud patterns and behaviors.
* **Latency**: Fraud prevention systems need to execute fast enough to be able to be deployed as part of a real-time payment approval process. Legacy systems often lack this capability.
* **Difficulty to change**: Legacy systems have been designed to work within specific parameters, and changing them to meet new requirements is often difficult and time-consuming.
* **Weak security**: Legacy systems may have outdated security protocols that leave organizations vulnerable to cyber attacks.
* **Operational overheads due to technical sprawl**: Existing architectures often pose operational challenges due to diverse technologies that have been deployed to support the different access patterns required by fraud models and ML training. This technical sprawl in the environment requires significant resources to maintain and update.
* **High operation costs**: Legacy systems can be costly to operate, requiring significant resources to maintain and update.
* **No collaboration between application and data science teams**: Technical boundaries between the operational platform and the data science platform are stopping application developers and data science teams from working collaboratively, leading to longer time to market and higher overheads.

These data issues can be detrimental to a financial institution trying desperately to keep up with the demands of customer expectations, user experience, and fraud. As technology is advancing rapidly, surely so is card fraud, becoming increasingly sophisticated. This has naturally led to an absolute need for real-time solutions to detect and prevent card fraud effectively. Anything less than that is unacceptable. So, how can financial institutions today meet these demands? The answer is simple. Fraud detection big data analytics should [shift-left to the application itself](https://www.mongodb.com/collateral/application-driven-analytics). 

What does this look like in practice? Application-driven analytics for fraud detection is the solution for the very real challenges financial institutions face today, as mentioned above.

## Solution overview
To break down what this looks like, we will demonstrate how easy it is to build an ML-based fraud solution using MongoDB and Databricks. The functional and nonfunctional features of this proposed solution include: 

* **Data completeness**: To address the challenge of incomplete data, the system will be integrated with external data sources to ensure complete and accurate data is available for analysis.
* **Real-time processing**: The system will be designed to process data in real time, enabling the timely detection of fraudulent activities.
* **AI/ML modeling and model use**: Organizations can leverage AI/ML to enhance their fraud prevention capabilities. AI/ML algorithms can quickly identify and flag potential fraud patterns and behaviors.
* **Real-time monitoring**: Organizations should aim to enable real-time monitoring of the application, allowing for real-time processing and analysis of data. 
* **Model observability**: Organizations should aim to improve observability in their systems to ensure that they have full visibility into fraud patterns and behaviors.
* **Flexibility and scalability**: The system will be designed with flexibility and [scalability](https://www.mongodb.com/collateral/scalability-with-mongodb-atlas) in mind, allowing for easy changes to be made to accommodate changing business needs and regulatory requirements.
* **Security**: The system will be designed with [robust security measures](https://www.mongodb.com/cloud/atlas/security) to protect against potential security breaches, including encryption, access control, and audit trails.
* **Ease of operation**: The system will be designed with ease of operation in mind, reducing operational headaches and enabling the fraud prevention team to focus on their core responsibilities..
* **Application development and data science team collaboration**: Organizations should aim to enable collaboration between application development and data science teams to ensure that the goals and objectives are aligned, and cooperation is optimized.
* **End-to-end CI/CD pipeline support**: Organizations should aim to have [end-to-end CI/CD pipeline](https://www.mongodb.com/docs/atlas/app-services/apps/cicd/) support to ensure that their systems are up-to-date and secure.

## Solution components
The functional features listed above can be implemented by a few architectural components. These include:

1. **Data sourcing** 
    1. **Producer apps**: The producer mobile app simulates the generation of live transactions. 
    2. **Legacy data source**: The SQL external data source is used for customer demographics.
    3. **Training data**: Historical transaction data needed for model training data is sourced from cloud object storage - Amazon S3 or Microsoft Azure Blob Storage. 
2. **MongoDB Atlas**: Serves as the Operational Data Store (ODS) for card transactions and processes transactions in real time. The solution leverages [MongoDB Atlas](https://www.mongodb.com/atlas) aggregation framework to perform in-app analytics to process transactions based on pre-configured rules and communicates with Databricks for advanced AI/ML-based fraud detection via a native Spark connector. 
3. **Databricks**: Hosts the AI/ML platform to complement [MongoDB Atlas in-app analytics](https://www.mongodb.com/use-cases/analytics). A fraud detection algorithm used in this example is a notebook inspired by [Databrick's fraud framework](https://www.databricks.com/solutions/accelerators/fraud-detection). MLFlow has been used to manage the MLOps for managing this model. The trained model is exposed as a REST endpoint. 

Now, let’s break down these architectural components in greater detail below, one by one.

![MongoDB for event-driven and shift-left analytics architecture](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/analytics_architecture_986d891aed.png)
***Figure 1**: MongoDB for event-driven and shift-left analytics architecture*

### 1. Data sourcing
The first step in implementing a comprehensive fraud detection solution is aggregating data from all relevant data sources. As shown in **Figure 1** above, an event-driven federated architecture is used to collect and process data from real-time sources such as producer apps, batch legacy systems data sources such as SQL databases, and historical training data sets from offline storage. This approach enables data sourcing from various facets such as transaction summary, customer demography, merchant information, and other relevant sources, ensuring data completeness. 

Additionally, the proposed event-driven architecture provides the following benefits: 
* Real-time transaction data unification, which allows for the collection of card transaction event data such as transaction amount, location, time of the transaction, payment gateway information, payment device information, etc., in **real-time**.
* Helps re-train monitoring models based on live event activity to combat fraud as it happens. 

The producer application for the demonstration purpose is a Python script that generates live transaction information at a predefined rate (transactions/sec, which is configurable).

![Figure 2: Transaction collection sample document](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/transaction_collection_sample_document_a4b9913a5b.png)
***Figure 2**: Transaction collection sample document*

### 2. MongoDB for event-driven, shift-left analytics architecture 
MongoDB Atlas is a managed data platform that offers several features that make it the perfect choice as the datastore for card fraud transaction classification. It supports [flexible data models](https://www.mongodb.com/basics/data-modeling) and can handle various types of data, high scalability to meet demand, advanced security features to ensure compliance with regulatory requirements, real-time data processing for fast and accurate fraud detection, and [cloud-based deployment](https://www.mongodb.com/cloud) to store data closer to customers and comply with local data privacy regulations.

The [MongoDB Spark Streaming Connector](https://www.mongodb.com/blog/post/introducing-mongodb-spark-connector-version-10-1) integrates Apache Spark and MongoDB. Apache Spark, hosted by Databricks, allows the processing and analysis of large amounts of data in real-time. The Spark Connector translates MongoDB data into Spark data frames and supports real time Spark streaming.

![Figure 3: MongoDB for event-driven and shift-left analytics architecture](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/event_driven_shift_left_analytics_architecture_4ee3a856dc.png)
***Figure 3**: MongoDB for event-driven and shift-left analytics architecture*

The [App Services features](https://www.mongodb.com/atlas/app-services) offered by MongoDB allow for real-time processing of data through change streams and triggers. Because MongoDB Atlas is capable of storing and processing various types of data as well as streaming capabilities and trigger functionality, it is well suited for use in an event-driven architecture. 

In the demo, we used both the rich connector ecosystem of [MongoDB and App Services](https://www.mongodb.com/docs/atlas/app-services/triggers/#:~:text=App%20Services%20supports%20three%20types%20of%20triggers:%201,triggers%20execute%20functions%20according%20to%20a%20pre-defined%20schedule.) to process transactions in real time. The App Service Trigger function is used by invoking a REST service call to an AI/ML model hosted through the Databricks MLflow framework.

![Figure 4: The processed and “features of transaction” MongoDB sample document](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/sample_document_261cf3de97.png)
***Figure 4**: The processed and “features of transaction” MongoDB sample document*

![Figure 5: Processed transaction sample document](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/processed_transaction_0b68d589b3.png)
***Figure 5**: Processed transaction sample document*

**Note**: *A combined view of the collections, as mentioned earlier, can be visually represented using [**MongoDB Charts**](https://www.mongodb.com/docs/charts/) to help better understand and observe the changing trends of fraudulent transactions. For advanced reporting purposes, [materialized views](https://www.mongodb.com/docs/manual/core/materialized-views/) can help.*

The example solution manages rules-based fraud prevention by storing user-defined payment limits and information in a user settings collection, as shown below. This includes maximum dollar limits per transaction, the number of transactions allowed per day, and other user-related details. By filtering transactions based on these rules before invoking expensive AI/ML models, the overall cost of fraud prevention is reduced.

![fraud detection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/fraud_detection_cd053647b2.png)

### 3. Databricks as an AI/ML ops platform
Databricks is a powerful AI/ML platform to develop models for identifying fraudulent transactions. One of the key features of Databricks is the support of real-time analytics. As discussed above, real-time analytics is a key feature of modern fraud detection systems. 

Databricks includes MLFlow, a powerful tool for managing the end-to-end machine learning lifecycle. MLFlow allows users to track experiments, reproduce results, and deploy models at scale, making it easier to manage complex machine learning workflows. MLFlow offers model observability, which allows for easy tracking of model performance and debugging. This includes access to model metrics, logs, and other relevant data, which can be used to identify issues and improve the accuracy of the model over time. Additionally, these features can help in the design of modern fraud detection systems using AI/ML.

## Demo artifacts and high-level description

Workflows needed for processing and building models for validating the authenticity of transactions are done through the Databricks AI/ML platform. There are mainly two workflow sets to achieve this:

1: The **Streaming workflow**, which runs in the background continuously to consume incoming transactions in real-time using the MongoDB Spark streaming connector. Every transaction first undergoes data preparation and a feature extraction process; the transformed features are then streamed back to the MongoDB collection with the help of a Spark streaming connector. 

![Figure 7: Streaming workflow](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/streaming_workflow_e8bbcd3dde.png)
***Figure 7**: Streaming workflow*

2: The **Training workflow** is a scheduled process that performs three main tasks/notebooks, as mentioned below. This workflow can be either manually triggered or through the Git CI/CD (webhooks).

![Figure 8: Training workflow stages ](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/training_workflow_stages_d1049534af.png)
***Figure 8**: Training workflow stages*

>A step-by-step breakdown of how the example solution works can be accessed at this [GitHub repository](https://github.com/ashwin-gangadhar-mdb/bfsi-demo), and an [end-to-end solution demo is available](https://drive.google.com/file/d/1_6YDqa-G6f9koklkF4l-gUpWczeDHJiu/view?usp=share_link). 

## Conclusion
Modernizing legacy fraud prevention systems using MongoDB and Databricks can provide many benefits, such as improved detection accuracy, increased flexibility and scalability, enhanced security, reduced operational headaches, reduced cost of operation, early pilots and quick iteration, and enhanced customer experience.

Modernizing legacy fraud prevention systems is essential to handling the challenges posed by modern fraud schemes. By incorporating advanced technologies such as MongoDB and Databricks, organizations can improve their fraud prevention capabilities, protect sensitive data, and reduce operational headaches. With the solution proposed, organizations can take a step forward in their fraud prevention journey to achieve their goals. 

Learn more about how MongoDB can modernize your fraud prevention system, and [contact the MongoDB team](https://www.mongodb.com/industries/financial-services).

In this article, we'll demonstrate how easy it is to build an ML-based fraud solution using MongoDB and Databricks.