Pavel Duchovny

Search functionality is a critical component of many modern web applications. Providing users with relevant results based on their search queries and additional filters dramatically improves their experience and satisfaction with your app.

In this article, we'll go over an implementation of search functionality using OpenAI's GPT-4 model and MongoDB's 
[Atlas Vector search](https://www.mongodb.com/products/platform/atlas-vector-search).  We've created a request handler function that not only retrieves relevant data based on a user's search query but also applies additional filters provided by the user.

Enriching the existing documents data with embeddings is covered in our main [Vector Search Tutorial](https://www.mongodb.com/developer/products/atlas/semantic-search-mongodb-atlas-vector-search/). 

## Search in the Airbnb app context ##

Consider a real-world scenario where we have an Airbnb-like app. Users can perform a free text search for listings and also filter results based on certain criteria like the number of rooms, beds, or the capacity of people the property can accommodate.

To implement this functionality, we use MongoDB's full-text search capabilities for the primary search, and OpenAI's GPT-4 model to create embeddings that contain the semantics of the data and use Vector Search to find relevant results.

![Listing search app](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blte10d5dd9a9d93a80/64d24c66f2f2b92e58763b73/Screenshot_2023-07-26_at_15.41.43.png)

The code to the application can be found in the following [GitHub repository](https://github.com/mongodb-developer/listings-AI-vector-search).

## The request handler
For the back end, we have used [Atlas app services](https://www.mongodb.com/docs/atlas/app-services/introduction/) with a simple [HTTPS “GET” endpoint](https://www.mongodb.com/docs/atlas/app-services/data-api/custom-endpoints/).
![Endpoint](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3a33581704e0805a/64d4a9f93118682f1c119867/Screenshot_2023-07-13_at_15.26.33.png)

Our function is designed to act as a request handler for incoming search requests.
When a search request arrives, it first extracts the search terms and filters from the query parameters. If no search term is provided, it returns a random sample of 30 listings from the database.

If a search term is present, the function makes a POST request to OpenAI's API, sending the search term and asking for an embedded representation of it using a specific model. This request returns a list of “embeddings,” or vector representations of the search term, which is then used in the next step.

```javascript

// This function is the endpoint's request handler. 
// It interacts with MongoDB Atlas and OpenAI API for embedding and search functionality.
exports = async function({ query }, response) {
    // Query params, e.g. '?search=test&beds=2' => {search: "test", beds: "2"}
    const { search, beds, rooms, people, maxPrice, freeTextFilter } = query;

    // MongoDB Atlas configuration.
    const mongodb = context.services.get('mongodb-atlas');
    const db = mongodb.db('sample_airbnb'); // Replace with your database name.
    const listingsAndReviews = db.collection('listingsAndReviews'); // Replace with your collection name.

    // If there's no search query, return a sample of 30 random documents from the collection.
    if (!search || search === "") {
      return await listingsAndReviews.aggregate([{$sample: {size: 30}}]).toArray();
    }

    // Fetch the OpenAI key stored in the context values.
    const openai_key = context.values.get("openAIKey");

    // URL to make the request to the OpenAI API.
    const url = 'https://api.openai.com/v1/embeddings';

    // Call OpenAI API to get the embeddings.
    let resp = await context.http.post({
        url: url,
        headers: {
            'Authorization': [`Bearer ${openai_key}`],
            'Content-Type': ['application/json']
        },
        body: JSON.stringify({
            input: search,
            model: "text-embedding-ada-002"
        })
    });

    // Parse the JSON response
    let responseData = EJSON.parse(resp.body.text());

    // Check the response status.
    if(resp.statusCode === 200) {
        console.log("Successfully received embedding.");

        // Fetch a random sample document.
      

        const embedding = responseData.data[0].embedding;
        console.log(JSON.stringify(embedding))

        let searchQ = {
            "index": "default",
            "knnBeta": {
                "vector": embedding,
                "path": "doc_embedding",
                "k": 100
            }
        };

        // If there's any filter in the query parameters, add it to the search query.
        if (freeTextFilter){
          // Turn free text search using GPT-4 into filter
            const sampleDocs = await listingsAndReviews.aggregate([
            { $sample: { size: 1 }},
            { $project: {
                _id: 0,
                bedrooms: 1,
                beds: 1,
                room_type: 1,
                property_type: 1,
                price: 1,
                accommodates: 1,
                bathrooms: 1,
                review_scores: 1
            }}
        ]).toArray();
        
          const filter =  await context.functions.execute("getSearchAIFilter",sampleDocs[0],freeTextFilter );
          searchQ.knnBeta.filter = filter;
        }
        else if(beds || rooms) {
          let filter = { "compound" : { "must" : [ ] } } 
          
           if (beds) {
             filter.compound.must.push({"equals" : {"value" : parseInt(beds) , path :"beds"}})
           }
           if (rooms)
           {
             filter.compound.must.push({"equals" : {"value" : parseInt(rooms) , path :"bedrooms"}})
           }
            
            searchQ.knnBeta.filter = filter;
        }

        // Perform the search with the defined query and limit the result to 50 documents.
        let docs = await listingsAndReviews.aggregate([
            { "$search": searchQ },
            { $limit : 50 }
        ]).toArray();

        return docs;
    } else {
        console.error("Failed to get embeddings");
        return [];
    }
};
```
To cover the filtering part of the query, we are using embedding and building a filter query to cover the basic filters that a user might request — in the presented example, two rooms and two beds in each.

```js
 else if(beds || rooms) {
          let filter = { "compound" : { "must" : [ ] } } 
          
           if (beds) {
             filter.compound.must.push({"equals" : {"value" : parseInt(beds) , path :"beds"}})
           }
           if (rooms)
           {
             filter.compound.must.push({"equals" : {"value" : parseInt(rooms) , path :"bedrooms"}})
           }
            
            searchQ.knnBeta.filter = filter;
        }
```
## Calling OpenAI API
![AI Filter](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7354c8177bc31007/64d4abb190cfc6027ab942d2/Screenshot_2023-07-26_at_15.41.17.png)

Let's consider a more advanced use case that can enhance our filtering experience. In this example, we are allowing a user to perform a free-form filtering that can provide sophisticated sentences, such as, “More than 1 bed and rating above 91.”

We call the OpenAI API to interpret the user's free text filter and translate it into something we can use in a MongoDB query. We send the API a description of what we need, based on the document structure we're working with and the user's free text input. This text is fed into the GPT-4 model, which returns a JSON object with 'range' or 'equals' operators that can be used in a MongoDB search query.


### getSearchAIFilter function

```javascript
// This function is the endpoint's request handler. 
// It interacts with OpenAI API for generating filter JSON based on the input.
exports = async function(sampleDoc, search) {
    // URL to make the request to the OpenAI API.
    const url = 'https://api.openai.com/v1/chat/completions';

    // Fetch the OpenAI key stored in the context values.
    const openai_key = context.values.get("openAIKey");

    // Convert the sample document to string format.
    let syntDocs = JSON.stringify(sampleDoc);
    console.log(syntDocs);

    // Prepare the request string for the OpenAI API.
    const reqString = `Convert programmatic command to Atlas $search filter only for range and equals  JS:\n\nExample: Based on document structure {"siblings" : '...', "dob" : "..."} give me the filter of all people  born  2015 and siblings are 3 \nOutput: {"filter":{ "compound" : { "must" : [ {"range": {"gte": 2015, "lte" : 2015,"path": "dob"} },{"equals" : {"value" : 3 , path :"siblings"}}]}}} \n\n  provide the needed filter to accomodate ${search}, pick a path from structure ${syntDocs}. Need just the json object with a range or equal operators. No explanation. No 'Output:' string in response. Valid JSON.`;
    console.log(`reqString: ${reqString}`);

    // Call OpenAI API to get the response.
    let resp = await context.http.post({
        url: url,
        headers: {
            'Authorization': `Bearer ${openai_key}`,
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({
            model: "gpt-4",
            temperature: 0.1,
            messages: [
                {
                    "role": "system",
                    "content": "Output  filter json generator follow only provided rules"
                },
                {
                    "role": "user",
                    "content": reqString
                }
            ]
        })
    });

    // Parse the JSON response
    let responseData = JSON.parse(resp.body.text());

    // Check the response status.
    if(resp.statusCode === 200) {
        console.log("Successfully received code.");
        console.log(JSON.stringify(responseData));

        const code = responseData.choices[0].message.content;
        let parsedCommand = EJSON.parse(code);
        console.log('parsed' + JSON.stringify(parsedCommand));

        // If the filter exists and it's not an empty object, return it.
        if (parsedCommand.filter && Object.keys(parsedCommand.filter).length !== 0) {
            return parsedCommand.filter;
        }
        
        // If there's no valid filter, return an empty object.
        return {};

    } else {
        console.error("Failed to generate filter JSON.");
        console.log(JSON.stringify(responseData));
        return {};
    }
};
```

## MongoDB search and filters

The function then constructs a MongoDB search query using the embedded representation of the search term and any additional filters provided by the user. This query is sent to MongoDB, and the function returns the results as a response —something that looks like the following for a search of “New York high floor” and “More than 1 bed and rating above 91.”

```javascript
{$search:{
    "index": "default",
    "knnBeta": {
        "vector": embedding,
        "path": "doc_embedding",
    "filter" : { "compound" : { "must" : [ {"range" : {"gte" : 1 , path :"beds"}},{"range" : {"gte" : 91, path :"score"}}]}},
        "k": 100
    }
}

}
```

## Conclusion
This approach allows us to leverage the power of OpenAI's GPT-4 model to interpret free text input and MongoDB's full-text search capability to return highly relevant search results. The use of natural language processing and AI brings a level of flexibility and intuitiveness to the search function that greatly enhances the user experience.

Remember, however, this is an advanced implementation. Ensure you have a good understanding of how MongoDB and OpenAI operate before attempting to implement a similar solution. Always take care to handle sensitive data appropriately and ensure your AI use aligns with OpenAI's use case policy.

This article delves into the integration of search functionality in web apps using OpenAI's GPT-4 model and MongoDB's Atlas Vector search. By harnessing the capabilities of AI and database management, we illustrate how to create a request handler that fetches data based on user queries and applies additional filters, enhancing user experience.

Leveraging OpenAI and MongoDB Atlas for Improved Search Functionality

# Generating MQL Shell Commands Using OpenAI and New mongosh Shell

[OpenAI](https://beta.openai.com/) is a fascinating and growing AI platform sponsored by Microsoft, allowing you to digest text cleverly to produce AI content with stunning results considering how small the “learning data set” you actually provide is.

MongoDB’s Query Language (MQL) is an intuitive language for developers to interact with MongoDB Documents. For this reason, I wanted to put OpenAI to the test of quickly learning the MongoDB language and using its overall knowledge to build queries from simple sentences. The results were more than satisfying to me. Github is already working on a project called [Github copilot](https://copilot.github.com/) which uses the same OpenAI engine to code.

In this article, I will show you my experiment, including the game-changing capabilities of the new [MongoDB Shell](https://docs.mongodb.com/mongodb-shell/) (`mongosh`) which can extend scripting with npm modules integrations.

## What is OpenAI and How Do I Get Access to It?

OpenAI is a unique project aiming to provide an API for many AI tasks built mostly on Natural Language Processing today. You can read more about their projects in this [blog](https://openai.com/blog/openai-api/).

There are a variety of examples for its [text processing capabilities](https://beta.openai.com/examples).

If you want to use OpenAI, you will need to get a trial API key first by joining the [waitlist](https://share.hsforms.com/1Lfc7WtPLRk2ppXhPjcYY-A4sk30) on their [main](https://beta.openai.com/) page. Once you are approved to get an API key, you will be granted about $18 for three months of testing. Each call in OpenAI is  [billed](https://beta.openai.com/pricing) and this is something to consider when using in production. For our purposes, $18 is more than enough to test the most expensive engine named “davinci.”


Once you get the API key, you can use various [clients](https://beta.openai.com/docs/developer-quickstart/community-libraries) to run their AI API from your script/application. 

Since we will be using the new `mongosh` shell, I have used the
 [JS API](https://www.npmjs.com/package/openai-api).

## Preparing the mongosh to Use OpenAI

First, we need to [install](https://docs.mongodb.com/mongodb-shell/install/) the new shell, if you haven’t done it so far. On my Mac laptop, I just issued:

``` bash
brew install mongosh
```

Windows users should download the MSI installer from our [download page](https://www.mongodb.com/try/download/shell) and follow the Windows instructions.

Once my mongosh is ready, I can start using it, but before I do so, let’s install [OpenAI JS](https://www.npmjs.com/package/openai-api), which we will import in the shell later on:

``` bash
$ mkdir openai-test
$ cd openai-test
Openai-test $ npm i openai-api
```

I’ve decided to use the Questions and Answers pattern, in the form of `Q: <Question>` and `A: <Answer>`, provided to the [text to command](https://beta.openai.com/examples/default-text-to-command) completion API to provide the learning material about MongoDB queries for the AI engine. To better feed it, I placed the training questions and answers in a file called `AI-input.txt` and its content:

```
Q: What is the query syntax?
A: db.collection.find(<filter>, <projection> , <options>)
Q:  Query users collection for username with value "boy"
A: db.users.find({"username" : "boy"})
Q:  Query users collection for username with value "girl"A:  db.users.find({"username" : "girl"})
Q: Query users collection for username with age bigger than 16
A:  db.users.find({"age" : {$gt : 16}})n;
Q: Query author collection for username with value "boy"
A: db.authors.find({"username" : "boy"})
Q:Query author collection for age lower than 7
A: db.authors.find({"age" : {$lt : 7}});

Q:insert a json document into collection authors with username equal to "girl"
A: db.authors.insert({"username" : "girl"}, {"age" : 10, "gender" : "female"})
Q: insert many documents into users collections
A: db.users.insertMany([{username : "john doe"}, {username : "joe doe"}]);
Q: The following aggregation uses $geoNear to find documents with a location at most 2 meters from the center [ -73.99279 , 40.719296 ] and a category equal to Parks.
A: db.places.aggregate([{ $geoNear: {near: { type: "Point", coordinates: [ -73.99279 , 40.719296 ] },distanceField: "dist.calculated",maxDistance: 2, query: { category: "Parks" },includeLocs: "dist.location", spherical: true}}])
Q: How to build atlas $search text query?
A: db.collection.aggregate({$search: {"index": <index name> "text": {"query": "<search-string>","path": "<field-to-search>",  "fuzzy": <options>,"score": <options>  }  }})

Q: What is the aggregate syntax?
A: db.collection.aggregate([<stage1>,<stage2>,<stage3>], <projection> , <options>);
Q: aggregate users collection to calculate salary sum per user
A: db.users.aggregate([{$group : { _id : "$username" , salary_sum : { $sum : "$salary" }}}]);
Q: aggregate person collection to calculate salary sum per person
A: db.persons.aggregate([{$group : { _id : "$person" , salary_sum : { $sum : "$salary" }}}]);
Q: Lookup users and orders collection
A: db.users.aggregate([{$lookup : {from: 'orders', localField : "_id", foreignField : "userId", as : "result" }} ]);

Q: What is the update syntax?
A:db.collection.update(query, update, options)
Q:  How to edit collection sports where sportname is 'football' and match is 'england vs portugal' to score of '3-3' and date to current date?
A: db.sports.update({ sportname: "football", match: "england vs portugal"} , {$set : {score: "3-3" , date : new Date()}} })
Q: Query and atomically update collection zoo where animal is "bear" with a counter increment on eat field, if the data does not exist user upsert
A: db.zoo.findOneAndUpdate({animal : "bear"}, {$inc: { eat : 1 }} , {upsert : true})
```

We will use this file later in our code.

This way, the completion will be based on a similar pattern.

### Prepare Your Atlas Cluster



[MongoDB Atlas](https://www.mongodb.com/cloud/atlas),  the database-as-a-platform service, is a great way to have a running cluster in seconds with a sample dataset already there for our test. To prepare it, please use the following steps:

1. Create an Atlas account (if you don’t have one already) and use/start a cluster. For detailed steps, follow this [documentation](https://docs.atlas.mongodb.com/getting-started/).
2. [Load the sample data set](https://docs.atlas.mongodb.com/sample-data/).
3. [Get your connection string](https://docs.atlas.mongodb.com/connect-to-cluster/).

Use the copied connection string, providing it to the `mongosh` binary to connect to the pre-populated Atlas cluster with sample data. Then, switch to `sample_restaurants`
database.


``` js
mongosh "mongodb+srv://<u>:<p>@<atlas-uri>/sample_restaurants"
Using Mongosh : X.X.X
Using MongoDB:		X.X.X

For mongosh info see: https://docs.mongodb.com/mongodb-shell/

ATLAS atlas-ugld61-shard-0 [primary]> use sample_restaurants;
```

## Using OpenAI Inside the mongosh Shell

Now, we can build our `textToMql` function by pasting it into the `mongosh`. The function will receive a text sentence, use our generated OpenAI API key, and will try to return the best MQL command for it:

``` js
async function textToMql(query){

const OpenAI = require('openai-api');
const openai-client = new OpenAI("<YOUR-OPENAI-API-KEY>");

const fs = require('fs');

var data =  await  fs.promises.readFile('AI-input.txt', 'utf8');

const learningPath = data;

var aiInput =  learningPath + "Q:" + query + "\nA:";

 const gptResponse = await openai-client.complete({
     engine: 'davinci',
     prompt: aiInput,
     "temperature": 0.3,
     "max_tokens": 400,
     "top_p": 1,
     "frequency_penalty": 0.2,
     "presence_penalty": 0,
     "stop": ["\n"]
 });

 console.log(gptResponse.data.choices[0].text);
}
```

In the above function, we first load the OpenAI npm module and initiate a client with the relevant API key from OpenAI. 

``` js
const OpenAI = require('openai-api');
const openai-client = new OpenAI("<YOUR-OPENAI-API-KEY>");

const fs = require('fs');
```

The new shell allows us to import built-in and external [modules](https://docs.mongodb.com/mongodb-shell/write-scripts/#require-a-native-module) to produce an unlimited flexibility with our scripts.

Then, we read the learning data from our `AI-input.txt` file. Finally we add our `Q: <query>` input to the end followed by the `A:` value which tells the engine we expect an answer based on the provided learningPath and our query. 


This data will go over to an OpenAI API call:

``` js
 const gptResponse = await openai.complete({
     engine: 'davinci',
     prompt: aiInput,
     "temperature": 0.3,
     "max_tokens": 400,
     "top_p": 1,
     "frequency_penalty": 0.2,
     "presence_penalty": 0,
     "stop": ["\n"]
 });
```

The call performs a completion API and gets the entire initial text as a `prompt` and receives some additional parameters, which I will elaborate on:

* `engine`: OpenAI supports a few AI engines which differ in quality and purpose as a tradeoff for pricing. The “davinci” engine is the most sophisticated one, according to OpenAI, and therefore is the most expensive one in terms of billing consumption.
* `temperature`: How creative will the AI be compared to the input we gave it? It can be between 0-1. 0.3 felt like a down-to-earth value, but you can play with it.
* `Max_tokens`: Describes the amount of data that will be returned.
* `Stop`: List of characters that will stop the engine from producing further content. Since we need to produce MQL statements, it will be one line based and “\n” is a stop character.

Once the content is returned, we parse the returned JSON and print it with `console.log`.

### Lets Put OpenAI to the Test with MQL

Once we have our function in place, we can try to produce a simple query to test it:

``` js
Atlas atlas-ugld61-shard-0 [primary] sample_restaurants> textToMql("query all restaurants where cuisine is American and name starts with 'Ri'")
 db.restaurants.find({cuisine : "American", name : /^Ri/})

Atlas atlas-ugld61-shard-0 [primary] sample_restaurants> db.restaurants.find({cuisine : "American", name : /^Ri/})
[
  {
    _id: ObjectId("5eb3d668b31de5d588f4292a"),
    address: {
      building: '2780',
      coord: [ -73.98241999999999, 40.579505 ],
      street: 'Stillwell Avenue',
      zipcode: '11224'
    },
    borough: 'Brooklyn',
    cuisine: 'American',
    grades: [
      {
        date: ISODate("2014-06-10T00:00:00.000Z"),
        grade: 'A',
        score: 5
      },
      {
        date: ISODate("2013-06-05T00:00:00.000Z"),
        grade: 'A',
        score: 7
      },
      {
        date: ISODate("2012-04-13T00:00:00.000Z"),
        grade: 'A',
        score: 12
      },
      {
        date: ISODate("2011-10-12T00:00:00.000Z"),
        grade: 'A',
        score: 12
      }
    ],
    name: 'Riviera Caterer',
    restaurant_id: '40356018'
  }
...
```

Nice! We never taught the engine about the  `restaurants` collection or how to filter with [regex](https://docs.mongodb.com/manual/reference/operator/query/regex/) operators but it still made the correct AI decisions. 

Let's do something more creative.

``` js
Atlas atlas-ugld61-shard-0 [primary] sample_restaurants> textToMql("Generate an insert many command with random fruit names and their weight")
 db.fruits.insertMany([{name: "apple", weight: 10}, {name: "banana", weight: 5}, {name: "grapes", weight: 15}])
Atlas atlas-ugld61-shard-0 [primary]sample_restaurants>  db.fruits.insertMany([{name: "apple", weight: 10}, {name: "banana", weight: 5}, {name: "grapes", weight: 15}])
{
  acknowledged: true,
  insertedIds: {
    '0': ObjectId("60e55621dc4197f07a26f5e1"),
    '1': ObjectId("60e55621dc4197f07a26f5e2"),
    '2': ObjectId("60e55621dc4197f07a26f5e3")
  }
}
```

Okay, now let's put it to the ultimate test: [aggregations](https://docs.mongodb.com/manual/aggregation/)!

``` js
Atlas atlas-ugld61-shard-0 [primary] sample_restaurants> use sample_mflix;
Atlas atlas-ugld61-shard-0 [primary] sample_mflix> textToMql("Aggregate the count of movies per year (sum : 1) on collection movies")
 db.movies.aggregate([{$group : { _id : "$year", count : { $sum : 1 }}}]);

Atlas atlas-ugld61-shard-0 [primary] sample_mflix>  db.movies.aggregate([{$group : { _id : "$year", count : { $sum : 1 }}}]);
[
  { _id: 1967, count: 107 },
  { _id: 1986, count: 206 },
  { _id: '2006è2012', count: 2 },
  { _id: 2004, count: 741 },
  { _id: 1918, count: 1 },
  { _id: 1991, count: 252 },
  { _id: 1968, count: 112 },
  { _id: 1990, count: 244 },
  { _id: 1933, count: 27 },
  { _id: 1997, count: 458 },
  { _id: 1957, count: 89 },
  { _id: 1931, count: 24 },
  { _id: 1925, count: 13 },
  { _id: 1948, count: 70 },
  { _id: 1922, count: 7 },
  { _id: '2005è', count: 2 },
  { _id: 1975, count: 112 },
  { _id: 1999, count: 542 },
  { _id: 2002, count: 655 },
  { _id: 2015, count: 484 }
]
```

Now *that* is the AI power of MongoDB pipelines!

## DEMO

[![asciicast](https://asciinema.org/a/424297.svg)](https://asciinema.org/a/424297)

## Wrap-Up

MongoDB's new shell allows us to script with enormous power like never before by utilizing npm external packages. Together with the power of OpenAI sophisticated AI patterns, we were able to teach the shell how to prompt text to accurate complex MongoDB commands, and with further learning and tuning, we can probably get much better results.

Try this today using the new MongoDB shell.

Learn how new mongosh external modules can be used to generate MQL language via OpenAI engine. Transform simple text sentences into sophisticated queries. 

Generating MQL Shell Commands Using OpenAI and New mongosh Shell

Modern platforms have a wide variety of data sources. As businesses grow, they have to constantly evolve their data management and have sophisticated, scalable, and convenient tools to analyse data from all sources to produce business insights.

MongoDB has developed a rich and powerful [query language](https://docs.mongodb.com/manual/tutorial/query-documents/), including a very robust [aggregation framework](https://docs.mongodb.com/manual/aggregation/). 

These were mainly done to optimize the way developers work with data and provide great tools to manipulate and query MongoDB documents.

Having said that, many developers, analysts, and tools still prefer the legacy SQL language to interact with the data sources. SQL has a strong foundation around joining data as this was a core concept of the legacy relational databases normalization model. 

This makes SQL have a convenient syntax when it comes to describing joins. 

Providing MongoDB users the ability to leverage SQL to analyse multi-source documents while having a flexible schema and data store is a compelling solution for businesses.

## Data Sources and the Challenge

Consider a requirement to create a single view to analyze data from operative different systems. For example:

- Customer data is managed in the user administration systems (REST API).
- Financial data is managed in a financial cluster (Atlas cluster).
- End-to-end transactions are stored in files on cold storage gathered from various external providers (cloud object storage - Amazon S3 or Microsoft Azure Blob Storage store).

How can we combine and best join this data? 

[MongoDB Atlas Query Federation](https://www.mongodb.com/docs/atlas/data-federation/) connects multiple data sources using the different [data store types](https://www.mongodb.com/docs/atlas/data-federation/overview/#mongodb-expression-exp.Data-Store). Once the data sources are mapped, we can create collections consuming this data. Those collections can have [SQL schema](https://docs.mongodb.com/datalake/reference/cli/sql/sqlgenerateschema/) generated, allowing us to perform sophisticated joins and do JDBC queries from various BI tools.

In this article, we will showcase the extreme power hidden in Atlas [SQL Query](https://www.mongodb.com/docs/atlas/data-federation/query/query-with-sql/).

## Setting Up My Federated Database Instance
In the following view, I have created three main data stores: 
- S3 Transaction Store (S3 sample data).
- Accounts from my Atlas clusters (Sample data sample_analytics.accounts).
- Customer data from a secure https source.

![Query Federation Sources](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_20_at_12_33_52_ec0802abed.png)

I mapped the stores into three collections under `FinTech` database:

- `Transactions`
- `Accounts`
- `CustomerDL`

Now, I can see them through a  Query Federation connection as MongoDB collections.

Let's grab our Query Federation instance connection string from the Atlas UI.

![Connection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_20_at_12_37_55_be8464c0d1.png)

This connection string can be used with our BI tools or client applications to run SQL queries.

## Connecting and Using $sql and db.sql

Once we connect to the Query Federation instancee via a mongosh shell, we can generate a SQL schema for our collections. This is optional for the JDBC or $sql operators to recognise collections as SQL “tables”  as this step is done automatically for newly created collections, however, its always good to be familiar with the available commands.

#### Generate SQL schema for each collection:
```js
use admin;
db.runCommand({sqlGenerateSchema: 1, sampleNamespaces: ["FinTech.customersDL"], sampleSize: 1000, setSchemas: true})
{
  ok: 1,
  schemas: [ { databaseName: 'FinTech', namespaces: [Array] } ]
}
db.runCommand({sqlGenerateSchema: 1, sampleNamespaces: ["FinTech.accounts"], sampleSize: 1000, setSchemas: true})
{
  ok: 1,
  schemas: [ { databaseName: 'FinTech', namespaces: [Array] } ]
}
db.runCommand({sqlGenerateSchema: 1, sampleNamespaces: ["FinTech.transactions"], sampleSize: 1000, setSchemas: true})
{
  ok: 1,
  schemas: [ { databaseName: 'FinTech', namespaces: [Array] } ]
}
```
#### Running SQL queries and joins using $sql stage:
```js
use FinTech;
db.aggregate([{
  $sql: {
    statement: "SELECT a.* , t.transaction_count FROM accounts a, transactions t where a.account_id = t.account_id SORT BY t.transaction_count DESC limit 2",
    format: "jdbc",
    formatVersion: 2,
    dialect: "mysql",
  }
}])

// Equivalent command
db.sql("SELECT a.* , t.transaction_count FROM accounts a, transactions t where a.account_id = t.account_id SORT BY t.transaction_count DESC limit 2");
```

The above query will prompt account information and the transaction counts of each account.

## Connecting Via JDBC

Let’s connect a powerful BI tool like Tableau with the [JDBC driver](https://www.mongodb.com/docs/atlas/data-federation/query/sql/drivers/jdbc/connect/).

[Download JDBC Driver](https://search.maven.org/artifact/org.mongodb/mongodb-jdbc).

![Latest Driver](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_20_at_12_44_11_e2df7beb3c.png)

#### Connect to Tableau
You have 2 main options to connect, via "MongoDB Atlas" connector or via a JDBC general connector. Please follow the relevant instructions and prerequisites on this [documentation](https://www.mongodb.com/docs/atlas/data-federation/query/sql/tableau/connect/) page.

##### Connector "MongoDB Atlas by MongoDB"
Search and click the  “MongoDB Atlas by MongoDB” connector and provide the information pointing to our Query Federation URI. See the following example:

![Tablue Connector](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_20_at_12_50_53_ace30bbb42.png)

##### "JDBC" Connector

Setting `connection.properties` file.
```
user=root
password=*******
authSource=admin
database=FinTech
ssl=true
compressors=zlib
```

Click the “Other Databases (JDBC)” connector, copy JDBC connection format, and load the `connection.properties` file.

![Tableau Connector Selection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/tableau_select_other_db_1c75d7023c.png)

Once the data is read successfully, the collections will appear on the right side.

#### Setting and Joining Data

![Setting Tables](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_11_19_53_8913b663c6.png)

We can drag and drop collections from different sources and link them together.

![Joining Tables with Drag & Drop](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_15_14_36_0e2fdcae97.png)

In my case, I connected `Transactions` => `Accounts` based on the `Account Id` field, and accounts and users based on the `Account Id` to `Accounts` field.

![Unified View of Accounts, Transactions and Users data](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_15_06_50_9e906438d5.png)

In this view, we will see a unified table for all accounts with usernames and their transactions start quarter. 

## Summary

MongoDB has all the tools to read, transform, and analyse your documents for almost any use-case.  

Whether your data is in an Atlas operational cluster, in a service, or on cold storage like cloud object storage, Atlas Query Federation will provide you with the ability to join the data in real time. With the option to use powerful join SQL syntax and SQL-based BI tools like Tableau, you can get value out of the data in no time.

Try Atlas Query Federation with your BI tools and SQL today.

Learn how new SQL-based queries can power your Query Federation insights in minutes. Integrate this capability with powerful BI tools like Tableau to get immediate value out of your data. 

Atlas Query Federation SQL to Form Powerful Data Interactions

>As of June 2022, the functionality previously known as Atlas Data Lake is now named Atlas Data Federation. Atlas Data Federation’s functionality is unchanged and you can learn more about it [here](http://mongodb.com/atlas/data-federation). Atlas Data Lake will remain in the Atlas Platform, with newly introduced functionality that you can learn about [here](http://mongodb.com/atlas/data-lake).

Modern platforms have a wide variety of data sources. As businesses grow, they have to constantly evolve their data management and have sophisticated, scalable, and convenient tools to analyse data from all sources to produce business insights.

MongoDB has developed a rich and powerful [query language](https://docs.mongodb.com/manual/tutorial/query-documents/), including a very robust [aggregation framework](https://docs.mongodb.com/manual/aggregation/). 

These were mainly done to optimize the way developers work with data and provide great tools to manipulate and query MongoDB documents.

Having said that, many developers, analysts, and tools still prefer the legacy SQL language to interact with the data sources. SQL has a strong foundation around joining data as this was a core concept of the legacy relational databases normalization model. 

This makes SQL have a convenient syntax when it comes to describing joins. 

Providing MongoDB users the ability to leverage SQL to analyse multi-source documents while having a flexible schema and data store is a compelling solution for businesses.

## Data Sources and the Challenge

Consider a requirement to create a single view to analyze data from operative different systems. For example:

- Customer data is managed in the user administration systems (REST API).
- Financial data is managed in a financial cluster (Atlas cluster).
- End-to-end transactions are stored in files on cold storage gathered from various external providers (cloud object storage - Amazon S3 or Microsoft Azure Blob Storage).

How can we combine and best join this data? 

[MongoDB Atlas Data Lake](https://www.mongodb.com/atlas/data-lake) connects multiple data sources using the different [source types](https://docs.mongodb.com/datalake/deployment/deploy-adl/). Once the data sources are mapped, we can create collections consuming this data. Those collections can have [SQL schema](https://docs.mongodb.com/datalake/reference/cli/sql/sqlgenerateschema/) generated, allowing us to perform sophisticated joins and do JDBC queries from various BI tools.

In this article, we will showcase the extreme power hidden in the Data Lake [SQL interface](https://docs.mongodb.com/datalake/admin/query-with-sql/).

## Setting Up My Data Lake
In the following view, I have created three main data sources: 
- S3 Transaction Store (S3 sample data).
- Accounts from my Atlas clusters (Sample data sample_analytics.accounts).
- Customer data from a secure https source.

![Data Lake Configuration](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_15_22_06_3bc2c2c9e7.png)

I mapped the stores into three collections under `FinTech` database:

- `Transactions`
- `Accounts`
- `CustomerDL`

Now, I can see them through a data lake connection as MongoDB collections.

Let's grab our data lake connection string from the Atlas UI.

![Data Lake Connection String](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_06_at_11_40_43_5ced192e43.png)

This connection string can be used with our BI tools or client applications to run SQL queries.

## Connecting and Using $sql

Once we connect to the data lake via a mongosh shell, we can generate a SQL schema for our collections. This is required for the JDBC or $sql operators to recognise collections as SQL “tables.”

#### Generate SQL schema for each collection:
```js
use admin;
db.runCommand({sqlGenerateSchema: 1, sampleNamespaces: ["FinTech.customersDL"], sampleSize: 1000, setSchemas: true})
{
  ok: 1,
  schemas: [ { databaseName: 'FinTech', namespaces: [Array] } ]
}
db.runCommand({sqlGenerateSchema: 1, sampleNamespaces: ["FinTech.accounts"], sampleSize: 1000, setSchemas: true})
{
  ok: 1,
  schemas: [ { databaseName: 'FinTech', namespaces: [Array] } ]
}
db.runCommand({sqlGenerateSchema: 1, sampleNamespaces: ["FinTech.transactions"], sampleSize: 1000, setSchemas: true})
{
  ok: 1,
  schemas: [ { databaseName: 'FinTech', namespaces: [Array] } ]
}
```
#### Running SQL queries and joins using $sql stage:
```js
use FinTech;
db.aggregate([{
  $sql: {
    statement: "SELECT a.* , t.transaction_count FROM accounts a, transactions t where a.account_id = t.account_id SORT BY t.transaction_count DESC limit 2",
    format: "jdbc",
    formatVersion: 2,
    dialect: "mysql",
  }
}])
```

The above query will prompt account information and the transaction counts of each account.

## Connecting Via JDBC

Let’s connect a powerful BI tool like Tableau with the JDBC driver.

[Download JDBC Driver](https://search.maven.org/search?q=a:mongodb-jdbc).

![JDBC driver](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/download_jdbc_for_tableau_7d0e5cb7d6.png)

Setting `connection.properties` file.
```
user=root
password=*******
authSource=admin
database=FinTech
ssl=true
compressors=zlib
```

#### Connect to Tableau

Click the  “Other Databases (JDBC)” connector and load the connection.properties file pointing to our data lake URI.

![Tableau Connector Selection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/tableau_select_other_db_1c75d7023c.png)

Once the data is read successfully, the collections will appear on the right side.

#### Setting and Joining Data

![Setting Tables](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_11_19_53_8913b663c6.png)

We can drag and drop collections from different sources and link them together.

![Joining Tables with Drag & Drop](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_15_14_36_0e2fdcae97.png)

In my case, I connected `Transactions` => `Accounts` based on the `Account Id` field, and accounts and users based on the `Account Id` to `Accounts` field.

![Unified View of Accounts, Transactions and Users data](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_10_17_at_15_06_50_9e906438d5.png)

In this view, we will see a unified table for all accounts with usernames and their transactions start quarter. 

## Summary

MongoDB has all the tools to read, transform, and analyse your documents for almost any use-case.  

Whether your data is in an Atlas operational cluster, in a service, or on cold storage like cloud object storage, Atlas Data Lake will provide you with the ability to join the data in real time. With the option to use powerful join SQL syntax and SQL-based BI tools like Tableau, you can get value out of the data in no time.

Try Atlas Data Lake with your BI tools and SQL today.



Learn how new SQL-based syntax can power your data lake insights in minutes. Integrate this capability with powerful BI tools like Tableau to get immediate value out of your data. 

Atlas Data Lake SQL Integration to Form Powerful Data Interactions

In this blog series, we are trying to inspire you with some reactive [Realm trigger](https://docs.mongodb.com/realm/triggers) use cases. We hope these will help you bring your application pipelines to the next level.

Essentially, triggers are components in our Atlas projects/Realm apps that allow a user to define a custom function to be invoked on a specific event.

* **Database triggers:** We have triggers that can be triggered based on database events—like ``deletes``, ``inserts``, ``updates``, and ``replaces``—called [database triggers](https://docs.mongodb.com/realm/triggers/database-triggers/).
* **Scheduled triggers**: We can schedule a trigger based on a ``cron`` expression via [scheduled triggers](https://docs.mongodb.com/realm/triggers/scheduled-triggers/).
* **Authentication triggers**: These triggers are only relevant for Realm authentication. They are triggered by one of the Realm auth providers' authentication events and can be configured only via a Realm application.

Relationships are an important part of any data design. Relational databases use primary and foreign key concepts to form those relationships when normalizing the data schema. Using those concepts, it allows a “cascading'' delete, which means a primary key parent delete will delete the related siblings.

MongoDB allows you to form relationships in different ways—for example, by embedding documents or arrays inside a parent document. This allows the document to contain all of its relationships within itself and therefore it does the cascading delete out of the box. Consider the following example between a user and the assigned tasks of the user:

``` js
{
userId : "abcd",
username : "user1@example.com" 
Tasks : [ 
         { taskId : 1, 
           Details : ["write","print" , "delete"]
          },
       { taskId : 1, 
         Details : ["clean","cook" , "eat"]
  }
}
```

Delete of this document will delete all the tasks.

However, in some design cases, we will want to separate the data of the relationship into Parent and Sibling collections—for example, ``games`` collection holding data for a specific game including ids referencing a ``quests`` collection holding a per game quest. As amount of quest data per game can be large and complex, we’d rather not embed it in ``games`` but reference:

**Games collection**

``` js
{
    _id: ObjectId("60f950794a61939b6aac12a4"),
    userId: 'xxx',
    gameId: 'abcd-wxyz',
    gameName: 'Crash',
    quests: [
      {
        startTime: ISODate("2021-01-01T22:00:00.000Z"),
        questId: ObjectId("60f94b7beb7f78709b97b5f3")
      },
      {
        questId: ObjectId("60f94bbfeb7f78709b97b5f4"),
        startTime: ISODate("2021-01-02T02:00:00.000Z")
      }
    ]
  }
```

Each game has a quest array with a start time of this quest and a reference to the quests collection where the quest data reside.

**Quests collection**

``` js
{
  _id: ObjectId("60f94bbfeb7f78709b97b5f4"),
  questName: 'War of fruits ',
  userId: 'xxx',
  details: {
    lastModified: ISODate("2021-01-01T23:00:00.000Z"),
    currentState: 'in-progress'
  },
  progressRounds: [ 'failed', 'failed', 'in-progress' ]
},
{
  _id: ObjectId("60f94b7beb7f78709b97b5f3"),
  questName: 'War of vegetable ',
  userId: 'xxx',
  details: {
    lastModified: ISODate("2021-01-01T22:00:00.000Z"),
    currentState: 'failed'
  },
  progressRounds: [ 'failed', 'failed', 'failed' ]
}
```

When a game gets deleted, we would like to purge the relevant quests in a cascading delete. This is where the **Preimage** trigger feature comes into play.

## Preimage Trigger Option

The Preimage option allows the trigger function to receive a snapshot of the deleted/modified document just before the change that triggered the function. This feature is enabled by enriching the oplog of the underlying replica set to store this snapshot as part of the change.
Read more on our [documentation](https://docs.mongodb.com/realm/mongodb/trigger-preimages/).

In our case, we will use this feature to capture the parent deleted document full snapshot (games) and delete the related relationship documents in the sibling collection (quests).

## Building the Trigger

When we define the database trigger, we will point it to the relevant cluster and parent namespace to monitor and trigger when a document is deleted—in our case, ``GamesDB.games``.

![Trigger  Def](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Cascade_Trigger_Def_a8acf9292f.png)

To enable the “Preimage” feature, we will toggle Document Preimage to “ON” and specify our function to handle the cascade delete logic.

![Preimage Toggle](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Pre_Imag_Toggle_a7a058ecd2.png)

**deleteCascadingQuests - Function**

``` js
exports = async function(changeEvent) {

  // Get deleted document preImage using "fullDocumentBeforeChange"
  var deletedDocument = changeEvent.fullDocumentBeforeChange;


  // Get sibling collection "quests"
  const quests = context.services.get("mongodb-atlas").db("GamesDB").collection("quests");

  // Delete all relevant quest documents.
  deletedDocument.quests.map( async (quest) => {
     await quests.deleteOne({_id : quest.questId});
  })
};
```

As you can see, the function gets the fully deleted “games” document present in “changeEvent.fullDocumentBeforeChange” and iterates over the “quests” array. For each of those array elements, the function runs a “deleteOne” on the “quests” collection to delete the relevant quests documents.

## Deleting the Parent Document

Now let's put our trigger to the test by deleting the game from the “games” collection:
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Delete_Game_df2f0c096c.png)

Once the document was deleted, our trigger was fired and now the “quests” collection is empty as it had only quests related to this deleted game:
![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Odocs_d929e3c940.png)
Our cascade delete works thanks to triggers “Preimages.”

## Wrap Up

The ability to get a modified or deleted full document opens a new world of opportunities for trigger use cases and abilities. We showed here one option to use this new feature but this can be used for many other scenarios, like tracking complex document state changes for auditing or cleanup images storage using the deleted metadata documents.

We suggest that you try this new feature considering your use case and look forward to the next trick along this blog series.

Want to keep going? Join the conversation over at our [community forums](https://www.mongodb.com/community/forums/)!

In this article, we will show you how to use a preimage feature to perform cascading relationship deletes via a trigger - based on the deleted parent document.

Triggers Treats and Tricks: Cascade Document Delete Using Triggers Preimage



<div class="introduction">

## Developer Productivity

[MongoDB Atlas](https://www.mongodb.com/cloud/atlas) is changing the database industry standards when it comes to database provisioning, maintenance, and scaling, as it just works. However, even superheroes like Atlas know that with Great Power Comes Great Responsibility.

For this reason, Atlas provides Enterprise-grade security features for your clusters and a set of [user management roles](https://docs.atlas.mongodb.com/reference/user-roles/) that can be assigned to log in users or [programmatic API keys](https://docs.atlas.mongodb.com/configure-api-access/).

However, since the management roles were built for a wide use case of
our customers there are some customers who need more fine-grained
permissions for specific teams or user types. Although, at the moment
the management roles are predefined, with the help of a simple Realm
service and the programmatic API we can allow user access for very
specific management/provisioning features without exposing them to a
wider sudo all ability.

To better understand this scenario I want to focus on the specific use
case of database user creation for the application teams. In this
scenario perhaps each developer per team may need its own user and
specific database permissions. With the current Atlas user roles you
will need to grant the team a `Cluster Manager Role`, which allows them
to change cluster properties as well as pause and resume a cluster. In
some cases this power is unnecessary for your users.

> If you haven't yet set up your free cluster on [MongoDB Atlas](http://bit.ly/mongodbatlas), now is a great time to do so. You have all the instructions in this [blog post](/quickstart/free-atlas-cluster/).

</div>

<div class="content">

## Proposed Solution

Your developers will submit their requests to a pre-built service which
will authenticate them and request an input for the user description.
Furthermore, the service will validate the input and post it to the
Atlas Admin API without exposing any additional information or API keys.

The user will receive a confirmation that the user was created and ready
to use.

## Work Flow

To make the service more accessible for users I am using a form-based
service called [Typeform](https://www.typeform.com/), you can choose many other available form builders (e.g [Google Forms](https://www.google.com/forms/about/)). This form will gather the information and password/secret for the service authentication from the user and pass it to the Realm webhook which will perform the action.

![The user fills the form and security
information](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/article-building-service-based-atlas-management/DrawingFlow.png "The user fills the form and security information.")  


![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/article-building-service-based-atlas-management/FormUsers.png)

The input is an [Atlas Admin API user object](https://docs.atlas.mongodb.com/reference/api/database-users-create-a-user/#request) that we want to create, looking something like the following object:

``` javascript
{
  "databaseName": <DB>,
  "password": <PWD>,
  "roles": [...],
  "username":  <USERNAME>
}
```

For more information please refer to our Atlas Role Based Authentication
[documentation](https://docs.atlas.mongodb.com/security-add-mongodb-users/).

## Webhook Back End

This section will require you to use an existing Realm Application or
[build a new one](https://docs.mongodb.com/realm/procedures/create-realm-app/).

MongoDB Realm is a serverless platform and mobile database. In our case
we will use the following features:

-   [Realm webhooks](https://docs.mongodb.com/realm/services/configure/service-webhooks/)
-   [Realm context HTTP Module](https://docs.mongodb.com/realm/functions/context/index.html#context-http)
-   [Realm Values/Secrets](https://docs.mongodb.com/realm/values-and-secrets/)

You will also need to [configure an Atlas Admin API key](https://docs.atlas.mongodb.com/configure-api-access/) for the relevant Project and obtain it's Project Id. This can be done from your Atlas project url (e.g., `https://cloud.mongodb.com/v2/<PROJECT_ID>#clusters`).

The main part of the Realm application is to hold the Atlas Admin API keys and information as [private secure secrets](https://docs.mongodb.com/realm/values-and-secrets/define-a-secret/).

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/article-building-service-based-atlas-management/Values-secrets.png)

This is the webhook configuration that will call our Realm Function each
time the form is sent:

![The Webhook definition.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/article-building-service-based-atlas-management/ServiceWebbhok.png "The Webhook should be defined as above.")

The function below receives the request. Fetch the needed API
information and sends the Atlas Admin API command. The result of which is
returned to the Form.

``` javascript
// This function is the webhook's request handler.
exports = async function(payload, response) {
     // Get payload
     const body = JSON.parse(payload.body.text());

     // Get secrets for the Atlas Admin API
     const username = context.values.get("AtlasPublicKey");
     const password = context.values.get("AtlasPrivateKey");
     const projectID = context.values.get("AtlasGroupId");

     //Extract the Atlas user object description
     const userObject = JSON.parse(body.form_response.answers[0].text);

     // Database users post command
     const postargs = {
     scheme: 'https',
     host: 'cloud.mongodb.com',
     path: 'api/atlas/v1.0/groups/' + projectID + '/databaseUsers',
     username: username,
     password: password,
     headers: {'Content-Type': ['application/json'], 'Accept': ['application/json']},
     digestAuth:true,
     body: JSON.stringify(userObject)};

     var res = await context.http.post(postargs);
     console.log(JSON.stringify(res));

     // Check status of the user creation and report back to the user.
     if (res.statusCode == 201)
     {
       response.setStatusCode(200)
       response.setBody(`Successfully created ${userObject.username}.`);
     } else {
       // Respond with a malformed request error
       response.setStatusCode(400)
       response.setBody(`Could not create user ${userObject.username}.`);
     }
};
```

Once the webhook is set and ready we can use it as a webhook url input
in the Typeform configuration.

The Realm webhook url can now be placed in the Typform webhook section.
Now the submitted data on the form will be forwarded via Webhook
integration to our webhook:

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/article-building-service-based-atlas-management/Webhook.png)

To strengthen the security around our Realm app we can strict the
allowed domain for the webhook request origin. Go to Realm application
"Manage" - "Settings" \> "Allowed Request Origins":

![](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/article-building-service-based-atlas-management/AllowOrigins.png)

We can test the form now by providing an [Atlas Admin API user
object](https://docs.atlas.mongodb.com/reference/api/database-users-create-a-user/#request).

![Submitting the object will result in a created user in the respectful
Atlas
Project](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/article-building-service-based-atlas-management/RunningExample.png "Submitting the object will result in a created user in the respectful Atlas Project")

If you go to the Atlas UI under the Database Access tab you will see the
created user.

![Atlas Database Access users
UI](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/article-building-service-based-atlas-management/UserExample.png)

</div>

<div class="summary">

## Summary

Now our developers will be able to create users quickly without being
exposed to any unnecessary privileges or human errors.

The webhook code can be converted to a function that can be called from
other webhooks or triggers allowing us to build sophisticated controlled
and secure provisioning methods. For example, we can configure a
scheduled trigger that pulls any newly created clusters and continuously
provision any new required users for our applications or edit any
existing users to add the needed new set of permissions.

[MongoDB Atlas](https://www.mongodb.com/cloud/atlas) and [Realm](https://www.mongodb.com/realm) platforms can work in great synergy allowing us to bring our devops and development cycles to the
next level.

</div>


Learn how to build Service-Based Atlas Cluster Management webhooks/functionality with Atlas Admin API and MongoDB Realm.

Building Service-Based Atlas Cluster Management

## Introduction

Containerized applications are becoming an industry standard for virtualization. When we talk about managing those containers, [Kubernetes](https://kubernetes.io/) will probably be brought up extremely quickly.

Kubernetes is a known open-source system for automating the deployment, scaling, and management of containerized applications. Nowadays, all of the major cloud providers (AWS, Google Cloud, and Azure) have a managed Kubernetes offering to easily allow organizations to get started and scale their Kubernetes environments.

Not surprisingly, MongoDB Atlas also runs on all of those offerings to give your modern containerized applications the best database offering. However, ease of development might yield in missing some critical aspects, such as security and connectivity control to our cloud services.

In this article, I will guide you on how to properly secure your Kubernetes cloud services when connecting to MongoDB Atlas using the recommended and robust solutions we have.

## Prerequisites

You will need to have a cloud provider account and the ability to deploy one of the Kubernetes offerings:

* [Amazon EKS](https://aws.amazon.com/eks/)
* [Google Cloud GKE](https://cloud.google.com/kubernetes-engine)
* [Azure AKS](https://azure.microsoft.com/en-us/services/kubernetes-service)

And of course, you'll need a MongoDB Atlas project where you are a project owner.

> If you haven't yet set up your free cluster on [MongoDB Atlas](http://bit.ly/mongodbatlas), now is a great time to do so. You have all the instructions in this [blog post](https://www.mongodb.com/developer/products/atlas/free-atlas-cluster/). Please note that for this tutorial you are required to have a M10+ cluster.

## Step 1: Set Up Networks

Atlas connections, by default, use credentials and end-to-end encryption to secure the connection. However, building a trusted network is a must for closing the security cycle between your application and the database.

No matter what cloud of choice you decide to build your Kubernetes cluster in, the basic foundation of securing that deployment is creating its own network. You can look into the following guides to create your own network and gather the main information (Names, Ids, and subnet [Classless Inter-Domain Routing](https://en.wikipedia.org/wiki/Classless_Inter-Domain_Routing) \- CIDR\)\.

##### Private Network Creation

| AWS | GCP | Azure |
| --- | --- | ----- |
| [Create an AWS VPC](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-getting-started.html#getting-started-create-vpc) | [Create a GCP VPC](https://cloud.google.com/vpc/docs/using-vpc) | [Create a VNET](https://docs.microsoft.com/en-us/azure/virtual-network/quick-create-portal) |

## Step 2: Create Network Peerings

Now, we'll configure connectivity of the virtual network that the Atlas region resides in to the virtual network we've created in Step 1. This connectivity is required to make sure the communication between networks is possible. We'll configure Atlas to allow connections from the virtual network from Step 1.

This process is called setting a Network Peering Connection. It's significant as it allows internal communication between networks of two different accounts (the Atlas cloud account and your cloud account).
<figure align="center"><img style="border-radius: 10px; width: 800px" src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Atlas_peering_page_3e33f7d799.png" alt="Atlas peering page"></figure>The network peerings are established under our Projects > Network Access > Peering > "ADD PEERING CONNECTION." For more information, please read our [documentation](https://docs.atlas.mongodb.com/security-vpc-peering/).

However, I will highlight the main points in each cloud for a successful peering setup:

##### Private Network Creation
<div style="overflow-x:auto;">
<table style="
        border: 1px solid #B8C4C2;
        border-collapse: collapse;
        margin: 16px 0;
        table-layout: fixed;
"><thead style="border: inherit" valign="bottom"><tr style="border: inherit"><th style="width: 250px; border: inherit; padding: 10px">AWS</th><th style="width: 250px; border: inherit; padding: 10px">GCP</th><th style="width: 250px; border: inherit; padding: 10px">Azure</th></tr></thead><tbody style="border: inherit" valign="top"><tr style="border: inherit"><td style="border: inherit; padding: 10px"><ol><li>Allow outbound traffic to Atlas CIDR on 2015-27017.</li><li>Obtain VPC information (<code>Account ID</code>, <code>VPC Name</code>, <code>VPC Region</code>, VPC CIDR). Enable DNS and Hostname resolution on that VPC.</li><li>Using this information, initiate the VPC Peering.</li><li>Approve the peering on AWS side.</li><li>Add peering route in the relevant subnet/s targeting Atlas CIDR and add those subnets/security groups in the Atlas access list page.</li></ol></td><td style="border: inherit; padding: 10px"><ol><li>Obtain GCP VPC information (<code>Project ID</code>, <code>VPC Name</code>, <code>VPC Region</code>, and CIDR).</li><li>When you initiate a VPC peering on Atlas side, it will generate information you need to input on GCP VPC network peering page (<code>Atlas Project ID</code> and <code>Atlas VPC Name</code>).</li><li>Submit the peering request approval on GCP and add the GCP CIDR in Atlas access lists.</li></ol></td><td style="border: inherit; padding: 10px"><ol><li>Obtain the following azure details from your subscription (<code>Subscription ID</code>, Azure Active Directory <code>Directory ID</code>, VNET <code>Resource Group Name</code>, <code>VNet Name</code>, <code>VNet Region</code>).</li><li>Input the gathered information and get a list of commands to perform on Azure console.</li><li>Open Azure console and run the commands, which will create a custom role and permissions for peering.</li><li>Validate and initiate peering.</li></ol></td></tr></tbody></table>
</div>

## Step 3: Deploy the Kubernetes Cluster in Our Networks

The Kubernetes clusters that we launch must be associated with the
peered network. I will highlight each cloud provider's specifics.

## AWS EKS

When we launch our EKS via the AWS console service, we need to configure
the peered VPC under the "Networking" tab.

Place the correct settings:

* VPC Name
* Relevant Subnets (Recommended to pick at least three availability
zones)
* Choose a security group with open 27015-27017 ports to the Atlas
CIDR.
* Optionally, you can add an IP range for your pods.

![In this case, I placed my peered VPC name, chose three availability zones from my EU West region, and specified a security group allowing 27015 - 27017 access to Atlas CIDR.](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/AWS_Netoworking_Kube_ed6404dca2.png)

## GCP GKE

When we launch our GKE service, we need to configure the peered VPC under the "Networking" section.

Place the correct settings:

* VPC Name
* Subnet Name
* Optionally, you can add an IP range for your pod's internal network that cannot overlap with the peered CIDR.

![In this case, once I got to the network tab, I placed my Network name ](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/GCP_Kube_Clusterr_Network_6037204a1a.png)

## Azure AKS

When we lunch our AKS service, we need to use the same resource group as the peered VNET and configure the peered VNET as the [CNI network](https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni) in the advanced Networking tab.

Place the correct settings:

* Resource Group
* VNET Name under "Virtual Network"
* Cluster Subnet should be the peered subnet range.
* The other CIDR should be a non-overlapping CIDR from the peered network.

![In this case, I checked the box ](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Networking_AKS_Azure_5c9cc20768.png)

## Step 4: Deploy Containers and Test Connectivity

Once the cluster is up and running in your cloud provider, you can test the connectivity to our peered cluster.

First, we will need to get our connection string and method from the Atlas cluster UI. Please note that GCP and Azure have private connection strings for peering, and those must be used for peered networks.

![Connect to Peering Test: step 1](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Private_Conn_String_d42ecbe933.png)

![Connect to Peering Test: step 2](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Shell_Template_e101dd97bf.png)

Now, let's test our connection from one of the Kubernetes pods:
<figure align="center"><img style="border-radius: 10px; width: 800px" src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Success_Shell_Private_3303fc8d3d.png" alt="Successful Shell Test"></figure>That's it. We are securely connected!

## Wrap-Up

Kubernetes-managed clusters offer a simple and modern way to deploy containerized applications to the vendor of your choice. It's great that we can easily secure their connections to work with the best cloud database offering there is, MongoDB Atlas, unlocking other possibilities such as building cross-platform application with [MongoDB Realm and Realm Sync](https://www.mongodb.com/cloud/atlas/lp/realm) or using [MongoDB Data Lake](https://www.mongodb.com/cloud/atlas/lp/data-lake) and [Atlas Search](https://www.mongodb.com/atlas/search) to build incredible applications.

> If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

A high-level guide on how to securely connect MongoDB Atlas with the Kubernetes offerings from Amazon AWS, Google Cloud (GCP), and Microsoft Azure.

Securely Connect MongoDB to Cloud-Offered Kubernetes Clusters

Harri Sarsa

Tutorial on Realm Serverless platform integration with the No-Code platform from AppGyver

Realm Serverless Platform Integration with No-Code Composer from AppGyver

Stanimira Vlaeva

Remix is a new and exciting javascript web framework. Together with the MongoDB Atlas Data API and Atlas Search it can form powerful web applications. A guided tour will show you how to leverage both technologies together. 

Next Gen Web Apps with Remix and MongoDB Atlas Data API



<div class="introduction">

In this blog series, we are trying to inspire you with some reactive [Realm trigger](https://docs.mongodb.com/realm/triggers/) use cases. We hope these will help you bring your application pipelines to the next level.

Essentially, triggers are components in our Atlas projects/Realm apps that allow a user to define a custom function to be invoked on a specific event.

-   **Database triggers:** We have triggers that can be scheduled based on database events—like `deletes`, `inserts`, `updates`, and `replaces`—called [database triggers](https://docs.mongodb.com/realm/triggers/database-triggers/).
-   **Scheduled triggers:** We can schedule a trigger based on a `cron` expression via [scheduled triggers](https://docs.mongodb.com/realm/triggers/scheduled-triggers/).
-   **Authentication triggers:** These triggers are only relevant for Realm authentication. They are triggered by one of the Realm auth providers' authentication events and can be configured only via a Realm application.

For this blog post, I would like to focus on trigger scheduling patterns.

Let me present a use case and we will see how the discussed mechanics might help us in this scenario. Consider a meeting management application that schedules meetings and as part of its functionality needs to notify a user 10 minutes before the meeting.

How would we create a trigger that will be fired 10 minutes before a timestamp which is only known by the "meeting" document?

First, let's have a look at the meetings collection documents example:

``` javascript
{
  _id : ObjectId("5ca4bbcea2dd94ee58162aa7"),
  event : "Mooz Meeting",
  eventDate : ISODate("2021-03-20:14:00:00Z"),
  meetingUrl : "https://mooz.meeting.com/5ca4bbcea2dd94ee58162aa7",
  invites : ["jon.doe@myemail.com", "doe.jonas@myemail.com"]
 }
```

I wanted to share an interesting solution based on triggers, and throughout this article, we will use a meeting notification example to explain the discussed approach.

## Prerequisites

First, verify that you have an Atlas project with owner privileges to create triggers.

-   [MongoDB Atlas account](https://docs.atlas.mongodb.com/tutorial/create-atlas-account/), [Atlas cluster](https://docs.atlas.mongodb.com/tutorial/deploy-free-tier-cluster/)
-   A [MongoDB Realm application](https://docs.mongodb.com/realm/procedures/create-realm-app/) or access to [MongoDB Atlas triggers](https://docs.atlas.mongodb.com/triggers/).

> If you haven't yet set up your free cluster on [MongoDB Atlas](http://bit.ly/mongodbatlas), now is a great time to do so. You have all the instructions in this [blog post](https://www.mongodb.com/developer/quickstart/free-atlas-cluster/).

## The Idea Behind the Main Mechanism

I will use the event example as a source document initiating the flow
with an insert to a `meetings` collection:

``` javascript
{
  _id : ObjectId("5ca4bbcea2dd94ee58162aa7"),
  event : "Mooz Meeting",
  eventDate : ISODate("2021-03-20:11:00:00Z"),
  meetingUrl : "https://mooz.meeting.com/5ca4bbcea2dd94ee58162aa7"
  invites : ["jon.doe@example.com"],
  phone : "+123456789"
}
```

Once we insert this document into the `meetings` collection, it will create the following record in a helper collection called `notifications` using an insert trigger:

``` javascript
{
  _id : ObjectId("5ca4bbcea2dd94ee58162aa7"),
  triggerDate : ISODate("2021-03-10:50:00:00Z")
}
```

The time and `_id` are calculated from the source document and aim to fire once `2021-03-10:50:00:00Z` arrives via a `fireScheduleTasks` trigger. This trigger is based on a delete operation out of a [TTL index](https://docs.mongodb.com/manual/core/index-ttl/) on the `triggerDate` field from the `notifications`.

This is when the user gets the reminder!

<figure>
<img src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/realm-triggers-treats-tricks/SMSMessage.jpg" class="align-center" />
</figure>

On a high level, here is the flow described above.

<figure>
<img src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/realm-triggers-treats-tricks/FlowSchemaDocumentBasedTrigger.png" class="align-center" alt="A meeting document is tracked by a trigger, creating a notification document. This document at the specified time will cause a delete event. The delete will fire a notification trigger to notify the user." /><figcaption  style="color: rgb(184, 196, 194); font-size: 12px; line-height: 18px;" >A meeting document is tracked by a trigger, creating a notification document. This document at the specified time will cause a delete event. The delete will fire a notification trigger to notify the user.</figcaption>
</figure>

There are three main components that allow our system to trigger based on our document data.

## 1. Define a Notifications Helper Collection

First, we need to prepare our `notifications` collection. This collection will be created implicitly by the following index creation command.

Now we will create a [TTL index](https://docs.mongodb.com/manual/core/index-ttl/). This index will cause the schedule document to expire when the value in `triggerDate` field arrives at its [expiry lifetime of 0 seconds](https://docs.mongodb.com/manual/tutorial/expire-data/#expire-documents-at-a-specific-clock-time) after its value.

``` javascript
db.notifications.createIndex( { "triggerDate": 1 }, { expireAfterSeconds: 0 } )
```

## 2. Building a Trigger to Populate the Schedule Collection

When setting up your `scheduleTasks` trigger, make sure you provide the following:

1.  Linked Atlas service and verify its name.
2.  The database and collection name we are basing the scheduling on, e.g., `meetings`.
3.  The relevant trigger operation that we want to schedule upon, e.g., when an event is inserted.
4.  Link it to a function that will perform the schedule collection population.

<figure>
<img src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/realm-triggers-treats-tricks/eventSchedulePublish.png" class="align-center" alt="My trigger UI configuration to populate the scheduling collection." /><figcaption  style="color: rgb(184, 196, 194); font-size: 12px; line-height: 18px;" >My trigger UI configuration to populate the scheduling collection.</figcaption>
</figure>

To populate the `notifications` collection with relevant triggering dates, we need to monitor our documents in the source collection. In our case, the user's upcoming meeting data is stored in the "meeting" collection with the userId field. Our trigger will monitor inserts to populate a Scheduled document.

``` javascript
exports = function(changeEvent) {
  // Get the notifications collection
  const coll = context.services.get("<ATLAS-SERVICE>").db("<DATABASE>").collection("notifications");

  // Calculate the "triggerDate" and populate the trigger collection and duplicate the _id
  const calcTriggerDate = new Date(changeEvent.fullDocument.eventDate - 10 * 60000);   
  return coll.insertOne({_id:changeEvent.fullDocument._id,triggerDate: calcTriggerDate });
};
```

>Important: Please replace \<ATLAS-SERVICE> and \<DATABASE> with your linked service and database names.

## 3. Building the Trigger to Perform the Action on the "Trigger Date"

To react to the TTL "delete" event happening exactly when we want our scheduled task to be executed, we need to use an "on delete" database trigger I call `fireScheduleTasks`.

When setting up your `fireScheduleTasks` trigger, make sure you provide the following:

1.  Linked Atlas service and verify its name.
2.  The database and collection for the notifications collection, e.g., `notifications`.
3.  The relevant trigger operation that we want to schedule upon, which is "DELETE."
4.  Link it to a function that will perform the fired task.

<figure>
<img src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/realm-triggers-treats-tricks/TaskSchedlueTrigger.png" class="align-center" />
</figure>

Now that we have populated the `notifications` collection with the `triggerDate`, we know the TTL index will fire a "delete" event with the relevant deleted `_id` so we can act upon our task.

In my case, 10 minutes before the user's event starts, my document will reach its lifetime and I will send a text using Twilio service to the attendee's phone.

A prerequisite for this stage will be to set up a [Twilio service](https://docs.mongodb.com/realm/services/twilio) using your Twilio cloud credentials.

1.  Make sure you have a [Twilio cloud account](https://www.twilio.com/login) with its SID and your Auth token.
2.  Set up the SID and Auth token into the Realm Twilio service configuration.
3.  Configure your [Twilio Messaging service](https://docs.mongodb.com/realm/services/twilio/#configure-twilio) and phone number.

Once we have it in place, we can use it to send SMS notifications to our invites.

``` javascript
exports = async function(changeEvent) {
  // Get meetings collection
  const coll = context.services.get("<ATLAS-SERVICE>").db("<DATABASE>").collection("meetings");

  // Read specific meeting document
  const doc = await coll.findOne({ _id: changeEvent.documentKey._id});

  // Send notification via Twilio SMS
  const twilio = context.services.get("<TWILIO-SERVICE>");
  twilio.send({
    to: doc.phone,
    from: "+123456789",
    body: `Reminder : Event ${doc.event} is about to start in 10min at ${doc.scheduledOn}`
    });
};
```

>Important: Replace \<ATLAS-SERVICE> and \<DATABASE> with your linked service and database names.

That's how the event was fired at the appropriate time.

## Wrap Up

With the presented technique, we can leverage existing triggering patterns to build new ones. This may open your mind to other ideas to design your next flows on MongoDB Realm.

In the following article in this series, we will learn how we can implement auto-increment with triggers.

> If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

In this article, we will explore a trick that lets us invoke a trigger task based on a date document field in our collections.

Realm Triggers Treats and Tricks - Document-Based Trigger Scheduling



<div class="introduction">

In this blog series, we are trying to inspire you with some reactive [Realm trigger](https://docs.mongodb.com/realm/triggers/) use cases. We hope these will help you bring your application pipelines to the next level.

Essentially, triggers are components in our Atlas projects/Realm apps that allow a user to define a custom function to be invoked on a specific event.

-   **Database triggers:** We have triggers that can be scheduled based on database events—like `deletes`, `inserts`, `updates`, and `replaces`—called [database triggers](https://docs.mongodb.com/realm/triggers/database-triggers/).
-   **Scheduled triggers:** We can schedule a trigger based on a `cron` expression via [scheduled triggers](https://docs.mongodb.com/realm/triggers/scheduled-triggers/).
-   **Authentication triggers:** These triggers are only relevant for Realm authentication. They are triggered by one of the Realm auth providers' authentication events and can be configured only via a Realm application.

For this blog post, I would like to showcase an auto-increment of a running ID in a collection similar to relational database sequence use. A sequence in relational databases like Oracle or SqlServer lets you use it to maintain a running ID for your table rows.

If we translate this into a `students` collection example, we would like to get the `studentId` field auto incremented.

``` javascript
{
  studentId : 1,
  studentName : "Mark Olsen",
  age : 15,
  phone : "+1234546789",
},
{
  studentId : 2,
  studentName : "Peter Parker",
  age : 17,
  phone : "+1234546788",
}
```

I wanted to share an interesting solution based on triggers, and throughout this article, we will use a students collection example with `studentsId` field to explain the discussed approach.

## Prerequisites

First, verify that you have an Atlas project with owner privileges to create triggers.

-   [MongoDB Atlas account](https://docs.atlas.mongodb.com/tutorial/create-atlas-account/), [Atlas cluster](https://docs.atlas.mongodb.com/tutorial/deploy-free-tier-cluster/)
-   A [MongoDB Realm application](https://docs.mongodb.com/realm/procedures/create-realm-app/) or access to [MongoDB Atlas triggers](https://docs.atlas.mongodb.com/triggers/).

> If you haven't yet set up your free cluster on [MongoDB Atlas](http://bit.ly/mongodbatlas), now is a great time to do so. You have all the instructions in this [blog post](https://www.mongodb.com/developer/quickstart/free-atlas-cluster/).
> 
## The Idea Behind the Main Mechanism

<figure>
<img src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/realm-triggers-treats-tricks/autoIncrementFlow.png" class="align-center" alt="The main flow of the auto-increment trigger concept" />
</figure>

Three main components allow our auto-increment mechanism to function.

## 1. Define a Source Collection

We should pick the collection that we need the auto-increment to work upon (`students`) and we can define a unique index on that field. This is not a must but it makes sense:

``` javascript
db.students.createIndex({studentsId : 1}, {unique : true});
```

## 2. Define a Generic Function to Auto-Increment the ID

In order for us to reuse the auto-increment code for more than one collection, I've decided to build a generic function and later associate it with the relevant triggers. Let's call the function `autoIncrement`. This function will receive an "insert" event from the source collection and increment a helper `counters` collection document that stores the current counter per collection. It uses `findOneAndUpdate` to return an automatically incremented value per the relevant source namespace, using the \_id as the namespace identifier. Once retrieved, the source collection is being set with a generic field called `<COLLECTION_NAME>Id` (in this example, `studentsId`).

```javascript
exports = async function(changeEvent) {
    // Source document _id
    const docId = changeEvent.fullDocument._id;

    // Get counter and source collection instances
    const counterCollection = context.services.get("<ATLAS-SERVICE>").db(changeEvent.ns.db).collection("counters");
    const targetCollection = context.services.get("<ATLAS-SERVICE>").db(changeEvent.ns.db).collection(changeEvent.ns.coll);

    // automically increment and retrieve a sequence relevant to the current namespace (db.collection)
    const counter = await counterCollection.findOneAndUpdate({_id: changeEvent.ns },{ $inc: { seq_value: 1 }}, { returnNewDocument: true, upsert : true});

    // Set a generic field <COLLECTION_NAME>Id 
    const doc = {};
    doc[`${changeEvent.ns.coll}Id`] = counter.seq_value;
    const updateRes = await targetCollection.updateOne({_id: docId},{ $set: doc});

    console.log(`Updated ${JSON.stringify(changeEvent.ns)} with counter ${counter.seq_value} result: ${JSON.stringify(updateRes)}`);
};
```

>Important: Replace \<ATLAS-SERVICE> with your linked service. The default value is "mongodb-atlas" if you have only one cluster linked to your Realm application.

Note that when we query and increment the counter, we expect to get the new version of the document `returnNewDocument: true` and `upsert: true` in case this is the first document.

The `counter` collection document after the first run on our student collection will look like this:

``` javascript
{
  _id: {
       db: "app",
       coll: "students"
  },
  seq_value: 1
}
```

## 3. Building the Trigger on Insert Operation and Associating it with Our Generic Function

Now let's define our trigger based on our Atlas cluster service and our database and source collection, in my case, `app.students`.

Please make sure to select "Event Ordering" toggled to "ON" and the "insert" operation.

<figure>
<img alt="trigger settings page" src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/realm-triggers-treats-tricks/autoIncrementTrigger1.png" class="align-center" />
</figure>

Now let's associate it with our pre-built function: `autoIncrement`.

<figure>
<img alt="prebuilt function selection page" src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/realm-triggers-treats-tricks/autoIncrementTrigger2.png" class="align-center" />
</figure>

Once we will insert our document into the collection, it will be automatically updated with a running unique number for `studentsId`.

<figure>
<img alt="inserted documents have the unique student id field" src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/realm-triggers-treats-tricks/studentsDocuments.png" class="align-center" />
</figure>

## Wrap Up

With the presented technique, we can leverage triggers to auto-increment and populate id fields. This may open your mind to other ideas to design your next flows on MongoDB Realm.

In the following article in this series, we will use triggers to auto-translate our documents and benefit from Atlas Search's multilingual abilities.

> If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.

In this article, we will explore a trick that lets us auto-increment a running ID using a trigger.

Triggers Treats and Tricks - Auto-Increment a Running ID Field

Data and event-driven applications are in high demand in a large variety of industries. With this demand, there is a growing challenge with how to sync the data across different data sources. 

A widely adopted solution for communicating real-time data transfer across multiple components in organization systems is implemented via clustered queues. One of the popular and proven solutions is [Apache Kafka](https://kafka.apache.org/).

The Kafka cluster is designed for streams of data that sequentially write events into commit logs, allowing real-time data movement between your services. Data is grouped into topics inside a Kafka cluster.

MongoDB provides a [Kafka connector](https://www.mongodb.com/kafka-connector) certified by [Confluent](https://www.confluent.io/), one of the largest Kafka providers. With the Kafka connector and Confluent software, you can publish data from a MongoDB cluster into Kafka topics using a source connector. Additionally, with a sink connector, you can consume data from a Kafka topic to persist directly and consistently into a MongoDB collection inside your MongoDB cluster.

In this article, we will provide a simple step-by-step guide on how to connect a remote Kafka cluster—in this case, a Confluent Cloud service—with a [MongoDB Atlas](https://mongodb.com/atlas) cluster. For simplicity purposes, the installation is minimal and designed for a small development environment. However, through the article, we will provide guidance and links for production-related considerations.

> **Pre-requisite**: To avoid JDK [known](https://docs.mongodb.com/drivers/java/sync/current/faq/#how-do-i-fix---javax.net.ssl.sslhandshakeexception--extension--5--should-not-be-presented-in-certificate_request--) certificate issues please update your JDK to one of the following patch versions or newer:
> - JDK 11.0.7+
> - JDK 13.0.3+
> - JDK 14.0.2+

## Table of Contents

1. [Create a Basic Confluent Cloud Cluster](#create-kafka-cluster)
1. [Create an Atlas Project and Cluster](#create-atlas-cluster)  
1. [Install Local Confluent Community Binaries to Run a Kafka Connect Instance](#install-kafka-connect)
1. [Configure the MongoDB Connector with Kafka Connect Locally](#configure-connectors)
1. [Start and Test Sink and Source MongoDB Kafka Connectors](#test-connectors)
1. [Summary](#summary)

## <a name="create-kafka-cluster"></a>Create a Basic Confluent Cloud Cluster

We will start by [creating](https://docs.confluent.io/cloud/current/clusters/create-cluster.html) a basic Kafka cluster in the [Confluent Cloud](https://confluent.cloud/).  

Once ready, create a topic to be used in the Kafka cluster. I created one named “orders.”

This “orders” topic will be used by Kafka Sink connector. Any data in this topic will be persisted automatically in the Atlas database.

![Kafka Cluster Topics](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/kafka_Topics_7eb45621e7.png)

You will also need another topic called "outsource.kafka.receipts". This topic will be used by the MongoDB Source connector, streaming reciepts from Atlas database.

[Generate](https://docs.confluent.io/cloud/current/access-management/authenticate/api-keys/api-keys.html) an `api-key` and `api-secret` to interact with this Kafka cluster. For the simplicity of this tutorial, I have selected the “Global Access” api-key. For production, it is recommended to give as minimum permissions as possible for the api-key used. Get a hold of the generated keys for future use.

Obtain the Kafka cluster connection string via `Cluster Overview > Cluster Settings > Identification > Bootstrap server` for future use. Basic clusters are open to the internet and in production, you will need to amend the access list for your specific hosts to connect to your cluster via advanced cluster ACLs.

## <a name="create-atlas-cluster"></a>Create a MongoDB Atlas Project and Cluster

[Create a project and cluster](https://docs.atlas.mongodb.com/getting-started/) or use an existing Atlas cluster in your project. 

![M0 Atlas Cluster](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_12_13_at_14_04_39_5fc276d2d6.png)

Prepare your Atlas cluster for a kafka-connect [connection](https://docs.mongodb.com/kafka-connector/current/introduction/connect/). Inside your project’s access list, enable user and relevant IP addresses of your local host, the one used for Kafka Connect binaries. Finally, get a hold of the Atlas [connection string](https://docs.atlas.mongodb.com/connect-to-cluster/#use-the-connect-to-your-cluster-dialog-to-connect-to-your-cluster) for future use.

## <a name="install-kafka-connect"></a>Install a Kafka Connect Worker

[Kafka Connect](https://docs.confluent.io/platform/current/connect) is one of the mechanisms to reliably stream data between different data systems and a Kafka cluster. For production use, we recommend using a [distributed deployment](https://docs.confluent.io/home/connect/self-managed/userguide.html#standalone-vs-distributed-mode) for high availability, fault tolerance, and scalability. There is also a cloud version to [install the connector](https://docs.confluent.io/cloud/current/connectors/cc-mongo-db-sink.html) on the Confluent Cloud.

For this simple tutorial, we will use a standalone local Kafka Connect [installation](https://docs.confluent.io/platform/current/installation/installing_cp/zip-tar.html).

To have the binaries to install kafka-connect and all of its dependencies, let’s download the files:
```shell 
curl -O http://packages.confluent.io/archive/7.0/confluent-community-7.0.1.tar.gz
tar -xvf confluent-community-7.0.1.tar.gz
```

## <a name="configure-connectors"></a>  Configure Kafka Connect

Configure the plugins directory where we will host the MongoDB Kafka Connector plugin:
```shell
mkdir -p /usr/local/share/kafka/plugins
```


Edit the `<confluent-install-dir>/etc/schema-registry/connect-avro-standalone.properties` using the content provided below. Ensure that you replace the `<kafka-cluster>:<kafka-port>` with information taken from Confluent Cloud bootstrap server earlier. 

Additionally, replace the generated `<api-key>` and `<api-secret>` taken from Confluent Cloud in every section.
```
bootstrap.servers=<kafka-cluster>:<kafka-port>

Connect data. Every Connect user will
# need to configure these based on the format they want their data in when loaded from or stored into Kafka
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
# Converter-specific settings can be passed in by prefixing the Converter's setting with the converter you want to apply
# it to
key.converter.schemas.enable=false
value.converter.schemas.enable=false

# The internal converter used for offsets and config data is configurable and must be specified, but most users will
# always want to use the built-in default. Offset and config data is never visible outside of Kafka Connect in this format.
internal.key.converter=org.apache.kafka.connect.json.JsonConverter
internal.value.converter=org.apache.kafka.connect.json.JsonConverter
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false

# Store offsets on local filesystem
offset.storage.file.filename=/tmp/connect.offsets
# Flush much faster than normal, which is useful for testing/debugging
offset.flush.interval.ms=10000

ssl.endpoint.identification.algorithm=https


sasl.mechanism=PLAIN
request.timeout.ms=20000
retry.backoff.ms=500
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="<api-key>" password="<api-secret>";
security.protocol=SASL_SSL

consumer.ssl.endpoint.identification.algorithm=https
consumer.sasl.mechanism=PLAIN
consumer.request.timeout.ms=20000
consumer.retry.backoff.ms=500
consumer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="<api-key>" password="<api-secret>";
consumer.security.protocol=SASL_SSL

producer.ssl.endpoint.identification.algorithm=https
producer.sasl.mechanism=PLAIN
producer.request.timeout.ms=20000
producer.retry.backoff.ms=500
producer.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
username="<api-key>" password="<api-secret>";
producer.security.protocol=SASL_SSL

plugin.path=/usr/local/share/kafka/plugins
```

**Important**: Place the `plugin.path`  to point to our plugin directory with permissions to the user running the kafka-connect process.

### Install the MongoDB connector JAR: 
Download the “[all](https://docs.mongodb.com/kafka-connector/current/introduction/install/#install-the-connector-on-confluent-platform)” jar and place it inside the plugin directory.

```shell
cp ~/Downloads/mongo-kafka-connect-1.6.1-all.jar /usr/local/share/kafka/plugins/
```
### Configure a MongoDB Sink Connector

The [MongoDB Sink connector](https://docs.mongodb.com/kafka-connector/current/sink-connector/) will allow us to read data off a specific Kafka topic and write to a MongoDB collection inside our cluster. Create a MongoDB sink connector properties file in the main working dir: `mongo-sink.properties` with your Atlas cluster details replacing `<username>:<password>@<atlas-cluster>/<database>` from your Atlas connect tab. The working directory can be any directory that the `connect-standalone` binary has access to and its path can be provided to the `kafka-connect` command shown in ["Start Kafka Connect and Connectors"](#test-connectors) section.

```
name=mongo-sink
topics=orders
connector.class=com.mongodb.kafka.connect.MongoSinkConnector
tasks.max=1
connection.uri=mongodb+srv://<username>:<password>@<atlas-cluster>/<database>?retryWrites=true&w=majority
database=kafka
collection=orders
max.num.retries=1
retries.defer.timeout=5000
```

With the above configuration, we will listen to the topic called “orders” and publish the input documents into database `kafka` and collection name `orders`. 

### Configure Mongo Source Connector

The [MongoDB Source connector](https://docs.mongodb.com/kafka-connector/current/source-connector/) will allow us to read data off a specific MongoDB collection topic and write to a Kafka topic. When data will arrive into a collection called `receipts`, we can use a source connector to transfer it to a Kafka predefined topic named “outsource.kafka.receipts” (the configured prefix followed by the `<database>.<collection>` name as a topic—it's possible to use advanced mapping to change that). 

Let’s create file `mongo-source.properties` in the main working directory:
```
name=mongo-source
connector.class=com.mongodb.kafka.connect.MongoSourceConnector
tasks.max=1

# Connection and source configuration
connection.uri=mongodb+srv://<username>:<password>@<atlas-cluster>/<database>?retryWrites=true&w=majority
database=kafka
collection=receipts

topic.prefix=outsource
topic.suffix=
poll.max.batch.size=1000
poll.await.time.ms=5000

# Change stream options
pipeline=[]
batch.size=0
change.stream.full.document=updateLookup
publish.full.document.only=true
collation=
```

The main properties here are the database, collection, and aggregation pipeline used to listen for incoming changes as well as the connection string. The `topic.prefix` adds a prefix to the `<database>.<collection>` namespace as the Kafka topic on the Confluent side. In this case, the topic name that will receive new MongoDB records is “outsource.kafka.receipts” and was predefined earlier in this tutorial.

I have also added `publish.full.document.only=true` as I only need the actual document changed or inserted without the change stream event wrapping information.

### Start Kafka Connect and Connectors

For simplicity reasons, I am running the standalone Kafka Connect in the foreground.

```
 ./confluent-7.0.1/bin/connect-standalone ./confluent-7.0.1/etc/schema-registry/connect-avro-standalone.properties mongo-sink.properties mongo-source.properties
```

> **Important**: Run with the latest Java version to avoid JDK SSL bugs.

Now every document that will be populated to topic “orders” will be inserted into the `orders` collection using a sink connector. A source connector we configured will transmit every receipt document from `receipt` collection back to another topic called "outsource.kafka.receipts" to showcase a MongoDB consumption to a Kafka topic.

## <a name="test-connectors"></a> Publish Documents to the Kafka Queue

Through the Confluent UI, I have submitted a test document to my “orders” topic.
![Produce data into "orders" topic](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_12_21_at_15_29_43_befe30feed.png)

### Atlas Cluster is Being Automatically Populated with the Data

Looking into my Atlas cluster, I can see a new collection named `orders` in the `kafka` database.

![Orders collection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_12_21_at_15_31_56_9a6f41a74d.png)

Now, let's assume that our application received the order document from the `orders` collection and produced a receipt. We can replicate this by inserting a document in the `kafka.reciepts` collection:

![Recipets collection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_12_21_at_15_33_22_0763e1d5a7.png)

This operation will cause the source connector to produce a message into “outsource.kafka.reciepts” topic.
### Kafka "outsource.kafka.reciepts" Topic

![Recieved data into "outsource.kafka.reciepts" topic](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screen_Shot_2021_12_21_at_15_33_56_7ab50fe570.png)

Log lines on kafka-connect will show that the process received and published the document: 

```
[2021-12-14 15:31:18,376] INFO [mongo-source|task-0] [Producer clientId=connector-producer-mongo-source-0] Cluster ID: lkc-65rmj (org.apache.kafka.clients.Metadata:287)
[2021-12-14 15:31:18,675] INFO [mongo-source|task-0] Opened connection [connectionId{localValue:21, serverValue:99712}] to dev-shard-00-02.uvwhr.mongodb.net:27017 (org.mongodb.driver.connection:71)
[2021-12-14 15:31:18,773] INFO [mongo-source|task-0] Started MongoDB source task (com.mongodb.kafka.connect.source.MongoSourceTask:203)
[2021-12-14 15:31:18,773] INFO [mongo-source|task-0] WorkerSourceTask{id=mongo-source-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:233)
[2021-12-14 15:31:27,671] INFO [mongo-source|task-0|offsets] WorkerSourceTask{id=mongo-source-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:505
[2021-12-14 15:31:37,673] INFO [mongo-source|task-0|offsets] WorkerSourceTask{id=mongo-source-0} flushing 1 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask:505)
```

## <a name="summary"></a>Summary

In this how-to article, I have covered the fundamentals of building a simple yet powerful integration of MongoDB Atlas to Kafka clusters using MongoDB Kafka Connector and Kafka Connect.

This should be a good starting point to get you going with your next event-driven application stack and a successful integration between MongoDB and Kafka.

Try out [MongoDB Atlas](https://mongodb.com/atlas) and [Kafka connector ](https://www.mongodb.com/kafka-connector) today!


A simple step-by-step tutorial on how to use MongoDB Atlas with a Kafka Connector and connect it to any Remote Kafka Cluster.

Kafka to MongoDB Atlas End to End Tutorial

Go is a modern language built on typed and native code compiling concepts while feeling and utilizing some benefits of dynamic languages. It is fairly simple to install and use, as it provides readable and robust code for many application use cases.

One of those use cases is building agents that report to a centralized data platform via streaming. A widely accepted approach is to communicate the agent data through subscription of distributed queues like Kafka. The Kafka topics can then propagate the data to many different sources, such as a [MongoDB Atlas](https://mongodb.com/atlas) cluster. 

Having a Go agent allows us to utilize the same code base for various operating systems, and the fact that it has good [integration with JSON data](https://www.mongodb.com/languages/golang) and packages such as a [MongoDB driver](https://docs.mongodb.com/drivers/go/) and [Confluent Go Kafka Client](https://docs.confluent.io/clients-confluent-kafka-go/current/overview.html) makes it a compelling candidate for the presented use case.

This article will demo how file size data on a host is monitored from a cross-platform agent written in Golang via a Kafka cluster using a Confluent hosted sink connector to MongoDB Atlas. MongoDB Atlas stores the data in a [time series collection](https://docs.mongodb.com/manual/core/timeseries-collections/). The [MongoDB Charts](https://www.mongodb.com/products/charts) product is a convenient way to show the gathered data to the user.
![Architecture Overview](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Mongo_DB_Dev_Hub_Article_Go_to_Mongo_DB_using_Kafka_Pavel_D_f06175feb5.jpg)
## Preparing the Golang project, Kafka cluster, and MongoDB Atlas

### Configuring a Go project

Our agent is going to run Go. Therefore, you will need to install the Go language software on your host.

Once this step is done, we will create a Go module to begin our project in our working directory:
``` shell
go mod init example/main
```
Now we will need to add the Confluent Kafka dependency to our Golang project:
``` shell
go get -u gopkg.in/confluentinc/confluent-kafka-go.v1/kafka
```

### Configuring a Kafka cluster 
Creating a Confluent Kafka Cluster is done via the Confluent UI. Start by [creating](https://docs.confluent.io/cloud/current/clusters/create-cluster.html) a basic Kafka cluster in the [Confluent Cloud](https://confluent.cloud/). Once ready, create a topic to be used in the Kafka cluster. I created one named “files.” 

[Generate](https://docs.confluent.io/cloud/current/access-management/authenticate/api-keys/api-keys.html) an api-key and api-secret to interact with this Kafka cluster. For the simplicity of this tutorial, I have selected the “Global Access” api-key. For production, it is recommended to give as minimum permissions as possible for the api-key used. Get a hold of the generated keys for future use.

Obtain the Kafka cluster connection string via Cluster Overview > Cluster Settings > Identification > Bootstrap server for future use. Basic clusters are open to the internet and in production, you will need to amend the access list for your specific hosts to connect to your cluster via advanced cluster ACLs.

> **Important:** The Confluent connector requires that the Kafka cluster and the Atlas cluster are deployed in the same region.
>
### Configuring Atlas project and cluster
[Create a project and cluster](https://docs.atlas.mongodb.com/getting-started/) or use an existing Atlas cluster in your project. 
![Atlas Cluster](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_01_31_at_13_58_17_fb16546b42.png)
Since we are using a time series collection, the clusters must use a 5.0+ version. Prepare your Atlas cluster for a Confluent sink Atlas [connection](https://docs.mongodb.com/kafka-connector/current/introduction/connect/). Inside your project’s access list, enable user and relevant IP addresses of your connector IPs. The access list IPs should be associated to the Atlas Sink Connector, which we will configure in a following section. Finally, get a hold of the Atlas [connection string](https://docs.atlas.mongodb.com/connect-to-cluster/#use-the-connect-to-your-cluster-dialog-to-connect-to-your-cluster) and the main cluster DNS. For more information about best securing and getting the relevant IPs from your Confluent connector, please read the following article: [MongoDB Atlas Sink Connector for Confluent Cloud](https://docs.confluent.io/cloud/current/connectors/cc-mongo-db-sink.html).

## Adding agent main logic
Now that we have our Kafka cluster and Atlas clusters created and prepared, we can initialize our agent code by building a small main file that will monitor my `./files` directory and capture the file names and sizes. I’ve added a file called `test.txt` with some data in it to bring it to ~200MB.

Let’s create a file named `main.go` and write a small logic that performs a constant loop with a 1 min sleep to walk through the files in the `files` folder:
``` go
package main

import (
   "fmt"
   "encoding/json"
   "time"
   "os"
   "path/filepath"
)

type Message struct {
  Name string
  Size float64
  Time int64
}

func samplePath (startPath string)  error {
   err := filepath.Walk(startPath,
    func(path string, info os.FileInfo, err error) error {
 
       var bytes int64
       bytes = info.Size()


       var kilobytes int64
       kilobytes = (bytes / 1024)

       var megabytes float64
       megabytes = (float64)(kilobytes / 1024)   // cast to type float64

       var gigabytes float64
       gigabytes = (megabytes / 1024)

      now := time.Now().Unix()*1000




      m := Message{info.Name(), gigabytes, now}
      value, err := json.Marshal(m)

      
      if err != nil {
         panic(fmt.Sprintf("Failed to parse JSON: %s", err))
      }


      fmt.Printf("value: %v\n", string(value))
      return nil;
  })
  if err != nil {
     return err
  }
  return nil;
}

func main() {
   for {
       err := samplePath("./files");
       if err != nil {
            panic(fmt.Sprintf("Failed to run sample : %s", err))
       }
       time.Sleep(time.Minute)
   }

}
```
The above code simply imports helper modules to traverse the directories and for JSON documents out of the files found. 

Since we need the data to be marked with the time of the sample, it is a great fit for time series data and therefore should eventually be stored in a time series collection on Atlas. If you want to learn more about time series collection and data, please read our article, [MongoDB Time Series Data](http://mongodb.com/features/mongodb-time-series-data).

We can test this agent by running the following command:
``` shell
go run main.go
```
The agent will produce JSON documents similar to the following format:
``` shell
value: {"Name":"files","Size":0,"Time":1643881924000}
value: {"Name":"test.txt","Size":0.185546875,"Time":1643881924000}
```
## Creating a Confluent MongoDB connector for Kafka
Now we are going to create a Kafka Sink connector to write the data coming into the “files” topic to our Atlas Cluster’s time series collection.

Confluent Cloud has a very popular integration running MongoDB’s Kafka connector as a hosted solution integrated with their Kafka clusters. Follow these [steps](https://docs.confluent.io/cloud/current/connectors/cc-mongo-db-sink.html#using-the-ccloud-console) to initiate a connector deployment.

The following are the inputs provided to the connector:
![Connector Setup 1](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_01_30_at_14_08_39_e1e3eff816.png)
![Connector Setup 2](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_01_30_at_14_08_54_6521f4b53e.png)
![Connector Setup 3](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_01_30_at_14_09_04_4de0d474ac.png)
Once you set it up,  following the guide, you will eventually have a similar launch summary page:
![Connector Final Summary](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_01_24_at_13_02_08_d460ad2de6.png)
After provisioning every populated document into the `files` queue will be pushed to a time series collection `hostMonitor.files` where the date field is `Time` and metadata field is `Name`.
## Pushing data to Kafka
Now let’s edit the `main.go` file to use a Kafka client and push each file measurement into the “files” queue.

Add the client library as an imported module:
``` go
import (
   "fmt"
   "encoding/json"
   "time"
   "os"
   "path/filepath"
   "github.com/confluentinc/confluent-kafka-go/kafka"
)
```
Add the Confluent cloud credentials and cluster DNS information. Replace `<CONFLUENT-SERVER>:<CONFLUENT-PORT>` found on the Kafka Cluster details page and the `<ACCESS-KEY>` , `<SECRET-KEY>` generated in the Kafka Cluster:
``` go
const (
   bootstrapServers = “<CONFLUENT-SERVER>:<CONFLUENT-PORT>"
   ccloudAPIKey     = "<ACCESS-KEY>"
   ccloudAPISecret  = "<SECRET-KEY>"
)
```
The following code will initiate the producer and produce a message out of the marshaled JSON document:    
``` go
               topic := "files"
              // Produce a new record to the topic...
               producer, err := kafka.NewProducer(&kafka.ConfigMap{
                   "bootstrap.servers":       bootstrapServers,
                   "sasl.mechanisms":         "PLAIN",
                   "security.protocol":       "SASL_SSL",
                   "sasl.username":           ccloudAPIKey,
                   "sasl.password":           ccloudAPISecret})

               if err != nil {
                   panic(fmt.Sprintf("Failed to create producer: %s", err))
               }

               producer.Produce(&kafka.Message{
                   TopicPartition: kafka.TopicPartition{Topic: &topic,
                       Partition: kafka.PartitionAny},
                   Value: []byte(value)}, nil)

               // Wait for delivery report
               e := <-producer.Events()

               message := e.(*kafka.Message)
               if message.TopicPartition.Error != nil {
                   fmt.Printf("failed to deliver message: %v\n",
                       message.TopicPartition)
               } else {
                   fmt.Printf("delivered to topic %s [%d] at offset %v\n",
                       *message.TopicPartition.Topic,
                       message.TopicPartition.Partition,
                       message.TopicPartition.Offset)
               }

               producer.Close()
 ```

The entire `main.go` file will look as follows:
``` go 
package main


import (
   "fmt"
   "encoding/json"
   "time"
   "os"
   "path/filepath"
   "github.com/confluentinc/confluent-kafka-go/kafka")

type Message struct {
  Name string
  Size float64
  Time int64
}


const (
   bootstrapServers = "<SERVER-DNS>:<SERVER-PORT>"
   ccloudAPIKey     = "<ACCESS-KEY>"
   ccloudAPISecret  = "<SECRET-KEY>"
)

func samplePath (startPath string)  error {
  
   err := filepath.Walk(startPath,
    func(path string, info os.FileInfo, err error) error {
    if err != nil {
            return err
    }
    fmt.Println(path, info.Size())

       var bytes int64
       bytes = info.Size()


       var kilobytes int64
       kilobytes = (bytes / 1024)

       var megabytes float64
       megabytes = (float64)(kilobytes / 1024)   // cast to type float64

       var gigabytes float64
       gigabytes = (megabytes / 1024)

      now := time.Now().Unix()*1000


      

       m := Message{info.Name(), gigabytes, now}
       value, err := json.Marshal(m)

      
       if err != nil {
         panic(fmt.Sprintf("Failed to parse JSON: %s", err))
       }


      fmt.Printf("value: %v\n", string(value))

               topic := "files"
               // Produce a new record to the topic...
                   producer, err := kafka.NewProducer(&kafka.ConfigMap{
                       "bootstrap.servers":       bootstrapServers,
                       "sasl.mechanisms":         "PLAIN",
                       "security.protocol":       "SASL_SSL",
                       "sasl.username":           ccloudAPIKey,
                       "sasl.password":           ccloudAPISecret})
  
                   if err != nil {
                       panic(fmt.Sprintf("Failed to create producer: %s", err))
                   }
  
                   producer.Produce(&kafka.Message{
                       TopicPartition: kafka.TopicPartition{Topic: &topic,
                           Partition: kafka.PartitionAny},
                       Value: []byte(value)}, nil)
  
                   // Wait for delivery report
                   e := <-producer.Events()
  
                   message := e.(*kafka.Message)
                   if message.TopicPartition.Error != nil {
                       fmt.Printf("failed to deliver message: %v\n",
                           message.TopicPartition)
                   } else {
                       fmt.Printf("delivered to topic %s [%d] at offset %v\n",
                           *message.TopicPartition.Topic,
                           message.TopicPartition.Partition,
                           message.TopicPartition.Offset)
                   }
  
                   producer.Close()

               return nil;
})
if err != nil {
       return err
}
   return nil;
}

func main() {
   for {
       err := samplePath("./files");
       if err != nil {
            panic(fmt.Sprintf("Failed to run sample : %s", err))
       }
     time.Sleep(time.Minute)
  
   }

}
```
Now when we run the agent while the Confluent Atlas sink connector is fully provisioned, we will see messages produced into the `hostMonitor.files` time series collection:
![Atlas Data](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_01_24_at_14_58_42_3f16961049.png)
## Analyzing the data using MongoDB Charts
To put our data into use, we can create some beautiful charts on top of the time series data. In a [line graph](https://docs.mongodb.com/charts/chart-type-reference/line-area-chart/#std-label-line-area-chart-ref), we configure the X axis to use the Time field, the Y axis to use the Size field, and the series to use the Name field. The following graph shows the colored lines represented as the evolution of the different file sizes over time.
![Line Graph Chart](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_01_24_at_13_00_53_0b3e2b9d25.png)
Now we have an agent and a fully functioning Charts dashboard to analyze growing files trends. This architecture allows big room for extensibility as the Go agent can have further functionalities, more subscribers can consume the monitored data and act upon it, and finally, MongoDB Atlas and Charts can be used by various applications and embedded to different platforms.

## Wrap Up
Building Go applications is simple yet has big benefits in terms of performance, cross platform code, and a large number of supported libraries and clients. Adding MongoDB Atlas via a Confluent Cloud Kafka service makes the implementation a robust and extensible stack, streaming data and efficiently storing and presenting it to the end user via Charts.

In this tutorial, we have covered all the basics you need to know in order to start using Go, Kafka, and MongoDB Atlas in your next streaming projects.

Try [MongoDB Atlas](https://www.mongodb.com/basics/mongodb-atlas-tutorial) and Go today!

Go is a cross-platform language. When combined with the power of Confluent Kafka streaming to MongoDB Atlas, you’ll be able to form tools and applications with real-time data streaming and analytics. Here’s a step-by-step tutorial to get you started.

Go to MongoDB Using Kafka Connectors - Ultimate Agent Guide

## Introduction

In recent years, "high availability" has been a buzzword in IT. Using this phrase usually means having your application and services resilient to any disruptions as much as possible.

As vendors, we have to guarantee certain levels of uptime via SLA contracts, as maintaining high availability is crucial to our customers.  These days, downtime, even for a short period of time, is widely unacceptable.

[MongoDB Atlas](https://www.mongodb.com/cloud/atlas), our data as a service platform, has just the right solution for you!

>
>
>If you haven't yet set up your free cluster on [MongoDB Atlas](http://bit.ly/mongodbatlas), now is a great time to do so. You have all the instructions in this [blog post](/quickstart/free-atlas-cluster).
>
>

## Let's Go Global

[MongoDB Atlas](https://www.mongodb.com/cloud/atlas) offers a very neat and flexible way to deploy a global database in the form of a [Global Sharded Cluster](https://docs.atlas.mongodb.com/global-clusters).  Essentially, you can create Zones across the world where each one will have a shard, essentially a replica set. This allows you to read and write data belonging to each region from its local shard/s.

To improve our network stability and overhead, Atlas provides a "Local reads in all Zones" button. It directs Atlas to automatically associate at least one secondary from each shard to one of the other regions. With an appropriate [read preference](https://docs.mongodb.com/manual/core/read-preference/), our application will now be able to get data from all regions without the need to query it cross-region. See our [Atlas Replica Set Tags](https://docs.atlas.mongodb.com/reference/replica-set-tags#std-label-replica-set-tags) to better understand how to target local nodes or specific cloud nodes.

MongoDB 4.4 introduced another interesting feature around read preferences for sharded clusters, called [Hedged Reads](https://docs.mongodb.com/manual/core/read-preference-hedge-option/#read-preference-hedged-read).  A hedged read query is run across two secondaries for each shard and returns the fastest response. This can allow us to get a fast response even if it is served from a different cloud member. Since this feature is allowed for `non-Primary` read preferences (like `nearest`), it should be considered to be eventually consistent. This should be taken into account with your [consistency considerations](https://docs.mongodb.com/manual/core/read-isolation-consistency-recency/index.html).

## Let's Go Multi-Cloud

One of the latest breakthroughs the Atlas service presented is being able to run a deployment across cloud vendors (AWS, Azure, and GCP). This feature is now available also in Global Clusters configurations.

We are now able to have shards spanning multiple clouds and regions, in one cluster, with one unified connection string. Due to the smart tagging of the replica set and hosts, we can have services working isolated within a single cloud, or benefit from being cloud agnostic.

To learn more about Multi-Cloud clusters, I suggest you read a great blog post, [Create a Multi-Cloud Cluster with MongoDB Atlas](https://developer.mongodb.com/how-to/setup-multi-cloud-cluster-mongodb-atlas), written by my colleague, Adrienne Tacke.

## What is the Insurance Policy We've Got?

![HA Check List](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/article/atlas-multi-cloud-global-clusters/Hachecklist.png)

When you set up a Global Cluster, how it is configured will change the availability features. As you configure your cluster, you can immediately see how your configuration covers your resiliency, HA, and Performance requirements. It's an awesome feature! Let's dive into the full set:

##### Zone Configuration Check-list

| Ability | Description | Feature that covers it |
| --- | --- | --- |
| <div style="width: 200px">Low latency read and writes in \<SHARD REGION></div> | Having a Primary in each region allows us to query/write data within the region. | Defining a zone in a region covers this ability. |
| Local reads in all zones | If we want to query a local node for another zone data (e.g., in America, query for Europe documents), we need to allow each other zone to place at least one secondary in the local region (e.g., Europe shard will have one secondary in America region). This requires our reads to use a latency based `readPreference` such as `nearest` or `hedged`. If we do not have a local node we will need to fetch the data remotely. | Pressing the "Allow local reads in all zones" will place one secondary in each other zone. |
| Available during partial region outage | In case there is a cloud "availability zone" outage within a specific region, regions with more than one availability zone will allow the region to still function as normal. | Having the preferred region of the zone with a number of electable nodes span across two or more availability zones of the cloud provider to withstand an availability zone outage. Those regions will be marked with a star in the UI. For example: two nodes in AWS N. Virginia where each one is, by design, deployed over three potential availability zones. |
| Available during full region outage | In case there is a full cloud region outage, we need to have a majority of nodes outside this region to maintain a primary within the | Having a majority of "Electable" nodes outside of the zone region. For example: two nodes in N. Virginia, two nodes in N. California, and one node in Ireland |
| Available during full cloud provider outage | If a whole cloud provider is unavailable, the zones still have a majority of electable nodes on other cloud providers, and so the zones are not dependent on one cloud provider. | Having multi-cloud nodes in all three clouds will allow you to withstand one full cloud provider failure. For example: two nodes on AWS N.Virginia, two nodes on GCP Frankfurt, and one node on Azure London. |


## Could the Apocalypse Not Scare Our Application?

After we have deployed our cluster, we now have a fully global cross-region, cross-cloud, fault-tolerant cluster with low read and write latencies across the globe. All this is accessed via a simple unified SRV connection string:

``` javascript
"mongodb+srv://user:myRealPassword@cluster0.mongodb.net/test?w=majority"
```

This cluster comes with a [full backup and point in time restore](https://docs.atlas.mongodb.com/backup/cloud-backup/overview) option, in case something **really** horrible happens (like human mistakes...).

I don't think that our application has anything to fear, other than its own bugs.

To show how easy it is to manipulate this complex deployment, I YouTubed it:

>
>
>:youtube[]{vid=pbhWjNVKMfg}
>
>To learn more about how to deploy a cross region global cluster to cover all of our fault tollerence best practices, check out the video.
>
>

## Wrap-Up

Covering our global application demand and scale has never been easier, while keeping the highest possible availability and resiliency. Global multi-cloud clusters allow IT to sleep well at night knowing that their data is always available, even in the apocalypse!

>
>
>If you have questions, please head to our [developer community website](https://community.mongodb.com/) where the MongoDB engineers and the MongoDB community will help you build your next big idea with MongoDB.
>
>


Learn how to build Atlas Multi-Cloud Global Cluster: Always available, even in the apocalypse!

Atlas Multi-Cloud Global Cluster: Always Available, Even in the Apocalypse!

The [document model](https://www.mongodb.com/document-databases) is the best way to work with data, and it’s the main factor to drive MongoDB popularity. The document model is also helping MongoDB innovate it's own solutions to power the world's most sophisticated data requirements.

[Data federation](https://www.mongodb.com/cloud/atlas/lp/data-federation), allows you to form federated database instances that span multiple data sources like different Atlas clusters, AWS S3 buckets, and other HTTPs sources. Now, one application or service can work with its individual cluster with dedicated resources and data compliance while queries can run on a union of the datasets. This is great for analytics or those global view dashboards and many other use cases in distributed systems.


[Atlas Search](https://www.mongodb.com/atlas/search) is also an emerging product that allows applications to build relevance-based search powered by Lucene directly on their MongoDB collections. While both products are amazing on their own, they can work together to form a multi-cluster, robust text search to solve challenges that were hard to solve beforehand.


## Example use case

Plotting attributes on a map based on geo coordinates is a common need for many applications. Complex code needs to be added if we want to merge different search sources into one data set based on the relevance or other score factors within a single request.

With Atlas federated queries run against Atlas search indexes, this task becomes as easy as firing one query. 

In my use case, I have two clusters: cluster-airbnb (Airbnb data) and cluster-whatscooking (restaurant data). For most parts of my applications, both data sets have nothing really in common and are therefore kept in different clusters for each application.

![Atlas Clusters for each application](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_27_at_11_00_13_798739a939.png)

However, if I am interested in plotting the locations of restaurants and Airbnbs (and maybe shops, later) around the user, I have to merge the datasets together with a search index built on top of the merged data. 

## With federated queries, everything becomes easier

As mentioned above, the two applications are running on two separated Atlas clusters due to their independent microservice nature. They can even be placed on different clouds and regions, like in this picture.

The restaurants data is stored in a collection named “restaurants” followed by a common modeling, such as grades/menu/location.

The Airbnb application stores a different data set model keeping Airbnb data, such as bookings/apartment details/location. 

The power of the document model and federated queries is that those data sets can become one if we create a federated database instance and group them under a “virtual collection” called “pointsOfInterest.”

![Atlas Query Federation setup](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_26_at_15_41_10_ada25a2f03.png)

The data sets can now be queried as if we have a collection named “pointsOfInterest” unioning the two.

## Lets add Atlas Search to the mix

Since the collections are located on Atlas, we can easily use Atlas search to individually index each. It’s also most probable that we already did that as our underlying applications require search capabilities of restaurants and Airbnb facilities. 

![Atlas Search index setup](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_26_at_14_38_50_3f9996155a.png)

However, if we make sure that the names of the indexes are identical—for example, “default”—and that key fields for special search—like geo—are the same (e.g., “location”), we can run federated search queries on “pointsOfInterest.” We are able to do that since the federated queries are propagated to each individual data source that comprise the virtual collection. With Atlas Search, it's surprisingly powerful as we can get results with a correct merging of the search scores between all of our data sets. This means that if geo search points of interest are close to my location, we will get either Airbnb or restaurants correctly ordered by the distance. What’s even cooler is that Atlas Data Federation intelligently “pushes down” as much of a query as possible, so the search operation will be done locally on the clusters and the union will be done in the federation layer, making this operation as efficient as possible.


![Compass geo query](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_26_at_15_03_36_fcb6d2d3e3.png)

## Finally, let's chart it up

We can take the query we just ran in Compass and export it to MongoDB Charts, our native charting offering that can directly connect to a federated database instance, plotting the data on a map:

:charts[]{url="https://charts.mongodb.com/charts-search-demos-rtbgg" id="62cea0c6-2fb0-4a7e-893f-f0e9a2d1ef39"}

## Wrap-up

With new products come new power and possibilities. Joining the forces of  [Data Federation](https://www.mongodb.com/docs/atlas/data-federation/) and [Atlas Search](https://www.mongodb.com/docs/atlas/atlas-search/) allows creators to easily form applications like never before. Start innovating today with  [MongoDB Atlas](https://www.mongodb.com/docs/atlas/getting-started/).


Atlas Data Federation opens a new world of data opportunities. Cross cluster search is available on MongoDB Atlas by combining the power of data federation on different Atlas Search indexes scattered cross different clusters, regions or even cloud providers.