Full Application

Snippet

MongoDB

Partner

Student

Community

Introductory

Advanced

Vishal Dhiman

Serverless solutions are quickly gaining traction among developers and organizations alike as a means to move fast, minimize overhead, and optimize costs. But shifting from a traditional pre-provisioned and predictable monthly bill to a consumption or usage-based model can sometimes result in confusion around how that bill is generated. In this article, we’ll take you through the basics of our serverless billing model and give you tips on how to best optimize your serverless database for cost efficiency.

## What are serverless instances?

MongoDB Atlas serverless instances, recently announced as [generally available](https://www.mongodb.com/blog/post/embrace-benefits-serverless-development-atlas), provide an on-demand serverless endpoint for your application with no sizing required. You simply choose a cloud provider and region to get started, and as your app grows, your serverless database will seamlessly scale based on demand and only charge for the resources you use.

Unlike our traditional clusters, serverless instances offer a fundamentally different [pricing model](https://www.mongodb.com/docs/atlas/billing/serverless-instance-costs/) that is primarily metered on reads, writes, and storage with automatic tiered discounts on reads as your usage scales. So, you can start small without any upfront commitments and never worry about paying for unused resources if your workload is idle.

### Serverless Database Pricing

Pay only for the operations you run.

| Item | Description | Pricing |
| ---- | ----------- | ------- |
| Read Processing Unit (RPU) | Number of read operations\* to the database *\*Sum of documents read divided by 4KB and index bytes read divided by 256 bytes* | $0.10/million for the first 50 million per day\* *\*Daily RPU tiers: Next 500 million: $0.05/million Reads thereafter: $0.01/million* |
| Write Processing Unit (WPU) | Number of write operations\* to the database \**Sum of documents and index bytes written, divided by 1KB* | $1.00/million |
| Storage | Data and indexes stored on the database | $0.25/GB-month |
| Standard Backup | Download and restore of backup snapshots\* \**2 free daily snapshots included per serverless instance* | $2.50/hour\* \**To download or restore the data* |
| Serverless Continuous Backup | 35-day backup retention for daily snapshots | $0.20/GB-month |
| Data Transfer | Inbound/outbound data to/from the database | $0.015 - $0.10/GB\* \**Depending on traffic source and destination* |

At first glance, read processing units (RPU) and write processing units (WPU) might be new units to you, so let’s quickly dig into what they mean. We use RPUs and WPUs to quantify the amount of work the database has to do to service a query, or to perform a write. To put it simply, a read processing unit (RPU) refers to the read operations to the database and one RPU is typically equivalent to either one document or one index entry scanned. Similarly, write processing units (WPUs) are write operations to the database, with one WPU typically being equivalent to each document or index entry written. For further explanation of cost units, please refer to our [documentation](https://www.mongodb.com/docs/atlas/billing/serverless-instance-costs/).

Now that you have a basic understanding of the pricing model, let’s go through an example to provide more context and tips on how to ensure your operations are best optimized to minimize costs.

For this example, we’ll be using the [sample dataset](https://www.mongodb.com/docs/atlas/sample-data/) in Atlas. To use sample data, simply go to your serverless instance deployment and select “Load Sample Dataset” from the dropdown as seen below.

![Load sample dataset](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/load_sample_dataset_6a05f77b85.png)
This will load a few collections, such as weather data and Airbnb listing data. Note that loading the sample dataset will consume approximately one million WPUs (less than $1 in most supported regions), and you will be [billed accordingly](https://www.mongodb.com/docs/atlas/billing/serverless-instance-costs/). 

Now, let’s take a look at what happens when we interact with our data and do some search queries.

## Scenario 1: Query on unindexed fields

For this exercise, I chose the sample\_weatherdata collection. While looking at the data in the Atlas Collections view, it’s clear that the weather data collection has information from various places and that most locations have a call letter code as a convenient way to identify where this weather reading data was taken.

For this example, let’s simulate what would happen if a user comes to your weather app and does a lookup by a geographic location. In this weather data collection, geographic locations can be identified by callLetters, which are specific codes for various weather stations across the world. I arbitrarily picked station code “ESVJ,” which is a weather buoy in the Atlantic Ocean. 

Here is what we see when we run this query in Atlas Data Explorer: 

![Document returned in sample_weatherdata.data collection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/document_returned_in_collection_e5267fb84f.png)

We can see this query returns three records. Now, let’s take a look at how many RPUs this query would cost me. Roughly, we should remember that:

RPU ≅ number of documents scanned. (In reality, if documents are larger than 4KB, then you might be charged more than 1 RPU per document.)

To execute the previous query, a full collection scan is required. Therefore, to run the query, all 10,000 documents in the collection are scanned, resulting in 10,000 RPUs (since each document is less than 4KB in length).

I took this query and ran this nearly 3,000 times through a shell script. This will simulate around 3,000 users coming to an app to check the weather in a day. Here is the code behind the script:

```
weatherRPUTest.sh

for ((i=0; i<=3000; i++)); do

    echo testing $i

    mongosh "mongodb+srv://vishalserverless1.qdxrf.mongodb.net/sample_weatherdata" --apiVersion 1 --username vishal --password ******** < mongoTest.js

done


mongoTest.js

db.data.find({callLetters: "ESVJ"})

```

As expected, 3,000 iterations will be 10,000 \* 3,000 = 30,000,000 RPUs = 30MM RPU = $3.00. 

You can see the invoice below:

![Atlas serverless invoice](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/atlas_serverless_invoice_18d2350015.png)

Based on this, the cost per user for this application would be $0.10 per user (calculated as: 30,000,000 / 3000 = 10,000 RPUs = $0.10). 

The cost of $0.10 per user seems to be very high for a database lookup, because if this weather app were to scale to reach a similar level of activity to [Accuweather](https://www.accuweather.com/en/press/43009943), who sees about 9.5B weather requests in a day, you’d be paying close to around $1 billion in database costs. By leaving your query this way, it’s likely that you’d be faced with an unexpectedly high bill as your usage scales—falling into a common trap that many new serverless users face. 

To avoid this problem, we recommend that you follow MongoDB best practices and [index your data](https://www.mongodb.com/docs/atlas/atlas-ui/indexes/) to optimize your queries for both performance and cost. [Indexes](https://www.mongodb.com/docs/manual/indexes/) are special data structures that store a small portion of the collection's data set in an easy-to-traverse form.

Without indexes, MongoDB must perform a collection scan—i.e., scan every document in a collection—to select those documents that match the query statement (something you just saw in the example above). By adding an index to appropriate queries, you can limit the number of documents it must inspect, significantly reducing the operations you are charged for.

Let’s look at how indexing can help you reduce your RPUs significantly.

## Scenario two: Querying with indexed fields

First, let’s create a simple index on the field ‘callLetters’:

![Create index on sample_weatherdata.data collection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/create_index_7844c36f97.png)

This operation will typically finish within 2-3 seconds. For reference, we can see the size of the index created on the index tab:

![Index size of callLetters_1 index](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/index_size_fcb8a1544e.png)

Due to the data structure of the index, the exact number of index reads are hard to compute. However, we can run the same script again for 3,000 iterations and again look at the invoice to compare.

![Atlas serverless invoice](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/invoice_276f68a5bf.png)

As seen on the invoice above, 3,000 queries on the indexed field did not change my bill at all, but you can see a slight increase in the RPUs (highlighted in blue). This is because the cost of running this operation was less than $0.01. Comparing the number of RPUs from the previous screenshot, we can see that they increased by 0.035M, which is around 35,000 RPUs, in contrast to the 30 million RPUs from the un-indexed query:

(30,000,000 - 35,000)/30,000,000 \* 100 = **99.88% reduction**

We can see that by simply adding the above index, we were able to reduce the cost per user to roughly $0.0001167 (calculated as: 35,000 / 3000 = 11.67 RPUs = $0.0001167), which is a huge cost savings compared to the previous cost of $0.10/user.

Therefore, indexing not only helps with improving the performance and scale of your queries, but it can also reduce your consumed RPUs significantly, which reduces your costs. Note that there can be rare scenarios where this is not true (where the size of the index is much larger than the number of documents). However, in most cases, you should see a significant reduction in cost and an improvement in performance.

## Take action to optimize your costs today

As you can see, adopting a usage-based pricing model can sometimes require you to be extra diligent in ensuring your data structure and queries are optimized. But when done correctly, the time spent to do those optimizations often pays off in more ways than one. 

If you’re unsure of where to start, we have [built-in monitoring tools](https://www.mongodb.com/docs/atlas/monitoring-alerts/) available in the Atlas UI that can help you. The [performance advisor](https://www.mongodb.com/docs/atlas/performance-advisor/#std-label-performance-advisor) automatically monitors your database for slow-running queries and will suggest new indexes to help improve query performance. Or, if you’re looking to investigate slow-running queries further, you can use [query profiler](https://www.mongodb.com/docs/atlas/tutorial/profile-database/#std-label-profile-database) to view a breakdown of all slow-running queries that occurred in the last 24 hours. If you prefer a terminal experience, you can also analyze your [query performance](https://www.mongodb.com/docs/manual/tutorial/analyze-query-plan/) in the MongoDB Shell or in MongoDB Compass. 

If you need further assistance, you can always contact our support team via chat or the [MongoDB support portal](https://support.mongodb.com/).

Shifting from a pre-provisioned to a serverless database can be challenging. Learn how to optimize your database and save money with these best practices.

Serverless Instances Billing 101: How to Optimize Your Bill with Indexing

Pavel Duchovny

The [document model](https://www.mongodb.com/document-databases) is the best way to work with data, and it’s the main factor to drive MongoDB popularity. The document model is also helping MongoDB innovate it's own solutions to power the world's most sophisticated data requirements.

[Data federation](https://www.mongodb.com/cloud/atlas/lp/data-federation), allows you to form federated database instances that span multiple data sources like different Atlas clusters, AWS S3 buckets, and other HTTPs sources. Now, one application or service can work with its individual cluster with dedicated resources and data compliance while queries can run on a union of the datasets. This is great for analytics or those global view dashboards and many other use cases in distributed systems.


[Atlas Search](https://www.mongodb.com/atlas/search) is also an emerging product that allows applications to build relevance-based search powered by Lucene directly on their MongoDB collections. While both products are amazing on their own, they can work together to form a multi-cluster, robust text search to solve challenges that were hard to solve beforehand.


## Example use case

Plotting attributes on a map based on geo coordinates is a common need for many applications. Complex code needs to be added if we want to merge different search sources into one data set based on the relevance or other score factors within a single request.

With Atlas federated queries run against Atlas search indexes, this task becomes as easy as firing one query. 

In my use case, I have two clusters: cluster-airbnb (Airbnb data) and cluster-whatscooking (restaurant data). For most parts of my applications, both data sets have nothing really in common and are therefore kept in different clusters for each application.

![Atlas Clusters for each application](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_27_at_11_00_13_798739a939.png)

However, if I am interested in plotting the locations of restaurants and Airbnbs (and maybe shops, later) around the user, I have to merge the datasets together with a search index built on top of the merged data. 

## With federated queries, everything becomes easier

As mentioned above, the two applications are running on two separated Atlas clusters due to their independent microservice nature. They can even be placed on different clouds and regions, like in this picture.

The restaurants data is stored in a collection named “restaurants” followed by a common modeling, such as grades/menu/location.

The Airbnb application stores a different data set model keeping Airbnb data, such as bookings/apartment details/location. 

The power of the document model and federated queries is that those data sets can become one if we create a federated database instance and group them under a “virtual collection” called “pointsOfInterest.”

![Atlas Query Federation setup](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_26_at_15_41_10_ada25a2f03.png)

The data sets can now be queried as if we have a collection named “pointsOfInterest” unioning the two.

## Lets add Atlas Search to the mix

Since the collections are located on Atlas, we can easily use Atlas search to individually index each. It’s also most probable that we already did that as our underlying applications require search capabilities of restaurants and Airbnb facilities. 

![Atlas Search index setup](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_26_at_14_38_50_3f9996155a.png)

However, if we make sure that the names of the indexes are identical—for example, “default”—and that key fields for special search—like geo—are the same (e.g., “location”), we can run federated search queries on “pointsOfInterest.” We are able to do that since the federated queries are propagated to each individual data source that comprise the virtual collection. With Atlas Search, it's surprisingly powerful as we can get results with a correct merging of the search scores between all of our data sets. This means that if geo search points of interest are close to my location, we will get either Airbnb or restaurants correctly ordered by the distance. What’s even cooler is that Atlas Data Federation intelligently “pushes down” as much of a query as possible, so the search operation will be done locally on the clusters and the union will be done in the federation layer, making this operation as efficient as possible.


![Compass geo query](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/Screenshot_2022_06_26_at_15_03_36_fcb6d2d3e3.png)

## Finally, let's chart it up

We can take the query we just ran in Compass and export it to MongoDB Charts, our native charting offering that can directly connect to a federated database instance, plotting the data on a map:

:charts[]{url="https://charts.mongodb.com/charts-search-demos-rtbgg" id="62cea0c6-2fb0-4a7e-893f-f0e9a2d1ef39"}

## Wrap-up

With new products come new power and possibilities. Joining the forces of  [Data Federation](https://www.mongodb.com/docs/atlas/data-federation/) and [Atlas Search](https://www.mongodb.com/docs/atlas/atlas-search/) allows creators to easily form applications like never before. Start innovating today with  [MongoDB Atlas](https://www.mongodb.com/docs/atlas/getting-started/).


Atlas Data Federation opens a new world of data opportunities. Cross cluster search is available on MongoDB Atlas by combining the power of data federation on different Atlas Search indexes scattered cross different clusters, regions or even cloud providers.

Cross Cluster Search Using Atlas Search and Data Federation

Jasiek Petryk

Here at [Scale Up](https://scaleup.com.pl/), we value music a lot. We have a Google Home speaker at our office that gets a lot of use. Music gets us going and helps us express ourselves individually as well as an organization. With how important music is in our lives, an idea came to our minds: We wanted to share what we like to listen to. We made [a Scale Up Spotify playlist](https://open.spotify.com/playlist/5oB8RrrUTu1Ulyw2995IcD?si=0a27142f344841aa) that we share on our website and listen to quite often, but we wanted to take it one step further. We wanted a way for others to be able to see what we're currently listening to in the office, and to host that, we turned to Atlas Application Services. 

Sources of music and ways to connect to the speaker are varied. Some people listen on YouTube, others on Spotify, some like to connect via a Cast feature that Google Home provides, and others just use Bluetooth connection to play the tunes. We sometimes use the voice features of Google Home and politely ask the speaker to put some music on.

All of this means that there's no easily available "one source of truth" for what's currently playing on the speaker. We could try to somehow connect to Spotify or Google Home's APIs to see what's being cast, but that doesn’t cover all the aforementioned cases—connecting via Bluetooth or streaming from YouTube. The only real source of truth is what our ears can actually hear. 

That's what we ultimately landed on—trying to figure out what song is playing by actually listening to soundwaves coming out of the speaker. Thankfully, there are [a lot of public APIs](https://rapidapi.com/collection/shazam-api) that can recognize songs based on a short audio sample. We decided to pick one that's pretty accurate when it comes to Polish music. In the end, it’s a big part of what we're listening to.

![Debugging application on an old and beaten up Nexus 5](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/debugging_application_nexus_5_2006fefe5e.png)

All of this has to run somewhere. The first thing that came to mind was to build this "listening device" by getting a Raspberry Pi with a microphone, but after going through my "old tech drawer"—let's face it, all of us techies have one—I found an old Nexus 5. After playing with some custom ROMs, I managed to run node.js applications there. If you think about it, it really is a perfect device for this use case. It has more than enough computing power, a built-in microphone, and a screen just in case you need to do a quick debug. I ended up writing a small program that takes a short audio sample every couple of minutes between 7:00 am and 5:00 pm and uses the API mentioned above to recognize the song.

The piece of information about what we're currently listening to is a good starting point, but in order to embed it on our website, we need to store it somewhere first. Here's where MongoDB's and Mongo Atlas' powers come into play. Setting up a cloud database was very easy. It took me less than five minutes. The free tier is more than enough for prototyping and simple use cases like this one, and if you end up needing more, you can always switch to a higher tier. I connected my application to a MongoDB Atlas instance using the [MongoDB Node Driver](https://www.mongodb.com/docs/drivers/node/current/).

Now that we have information about what's currently playing captured and safely stored in the MongoDB Atlas instance, there's only one piece of the puzzle missing: a way to retrieve the latest song from the database. Usually, this would require a separate application that we would have to develop, manage in the cloud, or provide a bare metal to run on, but here's the kicker: MongoDB has a way to do this easily with MongoDB Application Services. Application Services allows writing custom HTTP endpoints to retrieve or manipulate database data.

To create an endpoint like that, log in to your [MongoDB Atlas Account](https://account.mongodb.com/account/login). After creating a project, go to App Services at the top and then Create a New App. Name your app, click on Create App Service, and then on the left, you’ll see the HTTP Endpoints entry. After clicking Add Endpoint, select all the relevant settings. 

![Creating new Atlas Application HTTP Endpoint](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/creating_new_atlas_application_1fa3569e35.png)

The fetchsong function is a small JavaScript function that returns the latest song if the latest song had been played in the last 15 minutes and connected it to an HTTPS endpoint. Here it is in full glory: 

```Javascript
exports = async function (request, response) {
    const filter = {date: {$gt: new Date(new Date().getTime() - 15 * 60000)}};
    const projection = {artist: 1, title: 1, _id: 0};

    const songsCollection = context.services.get("mongodb-atlas")
        .db("scaleup")
        .collection("songs");
    const docs = await songsCollection
        .find(filter, projection)
        .sort({date: -1})
        .limit(1).toArray();

    const [latestSong] = docs;
    response.setBody(latestSong);
};
```

And voilà! After embedding a JavaScript snippet on our website to read song data here’s the final outcome:

![Currently played song on Spotify](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/currently_listening_on_spotify_958c61934e.png)

To see the results for yourself, visit [https://scaleup.com.pl/en/#music](https://scaleup.com.pl/en/#music). If you don't see anything, don’t worry—we work in the Central European Time Zone, so the office might be currently empty. :) Also, if you need to hire IT specialists here in Poland, don't hesitate to drop us a message. ;) 

Huge thanks to John Page for being an inspiration to play with MongoDB's products and to write this article. The source code for the whole project is [available on GitHub](https://github.com/janpetryk/scale-up-music-listener). :) 

Learn how Scale Up publishes the title of the song currently playing in their office regardless of the musical source using an old cellphone and MongoDB Atlas Application Services.

Articles

Featured Articles

Serverless Instances Billing 101: How to Optimize Your Bill with Indexing

Cross Cluster Search Using Atlas Search and Data Federation

Listen Along at Scale Up with Atlas Application Services

All Articles