Ken W. Alger

Anaiya Raisinghani

As an avid traveler, you have a love for staying at Airbnbs and have been keeping detailed notes about each one you’ve stayed in over the years. These notes are spread out across different storage locations, like MongoDB Atlas and AWS S3, making it a challenge to search for a specific Airbnb with the amenities your girlfriend desires for your upcoming Valentine’s Day trip. Luckily, there is a solution to make this process a lot easier. By using MongoDB’s Data Federation feature, you can combine all your data into one logical view and easily search for the perfect Airbnb without having to worry about where the data is stored. This way, you can make your Valentine’s Day trip perfect without wasting time searching through different databases and storage locations.

Don’t know how to utilize MongoDB’s Data Federation feature? This tutorial will guide you through exactly how to combine your Airbnb data together for easier query-ability.

## Tutorial Necessities

Before we jump in, there are a few necessities we need to have in order to be on the same page. This tutorial requires:

* MongoDB Atlas.
* An Amazon Web Services (AWS) account.
* Access to the AWS Management Console.
* AWS CLI.
* MongoDB Compass.

### Importing our sample data

Our first step is to import our Airbnb data into our Atlas cluster and our S3 bucket, so we have data to work with throughout this tutorial. Make sure to import the dataset into both of these storage locations.

### Importing via MongoDB Atlas

Step 1: Create a [free tier shared cluster](https://www.mongodb.com/basics/clusters/mongodb-cluster-setup).

Step 2: Once your cluster is set up, click the three ellipses and click “Load Sample Dataset."
![loading sample dataset](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/loading_sample_dataset_7433def6ca.png)

Step 3: Once you get this green message you’ll know your sample dataset (Airbnb notes) is properly loaded into your cluster.
![sample dataset successfull loaded](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/sample_dataset_successfully_loaded_3ad17e57bb.png)

### Importing via AWS S3
Step 1: We will be using [this sample data set](https://developer-advocacy-public.s3-eu-west-1.amazonaws.com/blog/sampledata.zip). Please download it locally. It contains the sample data we are working with along with the S3 bucket structure necessary for this demo. 

Step 2: Once the data set is downloaded, access your AWS Management Console and navigate to their S3 service. 

Step 3: Hit the button “Create Bucket” and [follow the instructions](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html) to create your bucket and upload the sampledata.zip. 

Step 4: Make sure to unzip your file before uploading the folders into S3. 

Step 5: Once your data is loaded into the bucket, you will see several folders, each with varying data types.

![folders with data types](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/folders_with_data_types_3c53eeeefa.png)

Step 6: Follow the path: Amazon S3 > Buckets > atlas-data-federation-demo > json/ > airbnb/ to view your Airbnb notes. Your bucket structure should look like this:

![bucket structure](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/bucket_structure_4852f173bc.png)

Congratulations! You have successfully uploaded your extensive Airbnb notes in not one but two storage locations. Now, let’s see how to retrieve this information in one location using Data Federation so we can find the perfect Airbnb. In order to do so, we need to get comfortable with the MongoDB Atlas Data Federation console. 

## Connecting MongoDB Atlas to S3

Inside the MongoDB Atlas console, on the left side, click on Data Federation.

![Data Federation](https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltc94edfdcc342265b/649c786b84a4c7eb46d8db62/data-federation.png)

Here, click “set up manually” in the "create new federated database" dropdown in the top right corner of the UI. This will lead us to a page where we can add in our data sources. You can rename your Federated Database Instance to anything you like. Once you save it, you will not be able to change the name.

![Naming your Data Federation instance](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/naming_data_federation_instance_65ee5be371.png)

Let’s add in our data sources from our cluster and our bucket!

### Adding in data source via AWS S3 Bucket:
Step 1: Click on “Add Data Source.” 

Step 2: Select the “Amazon S3” button and hit “Next.” 

![Adding a data source](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/adding_data_source_ef7ae8478d.png)

Step 3: From here, click Next on the “Authorize an AWS IAM Role”:

![Configure AWS S3 data store](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/configure_data_store_0c31660f41.png)

Step 4: Click on “Create New Role in the AWS CLI”: 

![Creating a new role](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/creating_a_new_role_4a772bcd89.png)

Step 5: Now, you’re going to want to make sure you have AWS CLI configured on your laptop. 

Step 6: Follow the steps below the “Create New Role with the AWS CLI” in your AWS CLI. 

```
aws iam create-role \
 --role-name datafederation \
 --assume-role-policy-document file://role-trust-policy.json
```

Step 7: You can find your “ARN” directly in your terminal. Copy that in — it should look like this:

```
arn:aws:iam::7***************:role/datafederation
```

Step 8: Enter the bucket name containing your Airbnb notes:

![Enter bucket name containing Airbnb notes](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/enter_bucket_name_2b78f110b3.png)

Step 9: Follow the instructions in Atlas and save your policy role. 

![Save policy role](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/save_policy_role_4315553ee6.png)

Step 10: Copy the CLI commands listed on the screen and paste them into your terminal like so:

```
aws iam put-role-policy \
  --role-name datafederation \
  --policy-name datafederation-policy \
  --policy-document file://adl-s3-policy.json
```

Step 11: Access your AWS Console, locate your listingsAndReviews.json file located in your S3 bucket, and copy the S3 URI. 

![Copy S3 URI](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/copy_s3_uri_e12ada2d17.png)

Step 12: Enter it back into your “Define ‘Data Sources’ Using Paths Inside Your S3” screen and change each step of the tree to “static.” 

![Changing steps to static](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/changing_steps_to_static_75b0b6ab43.png)

Step 13:  Drag your file from the left side of the screen to the middle where it says, “Drag the dataset to your Federated Database.” Following these steps correctly will result in a page similar to the screenshot below.

![Data source to virtual connection](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/data_source_to_virtual_connection_6e7e3a7252.png)

You have successfully added in your Airbnb notes from your S3 bucket. Nice job. Let's do the same thing for the notes saved in our Atlas cluster. 

### Adding in data source via MongoDB Atlas cluster
Step 1: Click “Add Data Sources.”

Step 2: Select “MongoDB Atlas Cluster” and provide the cluster name along with our sample_airbnb collection. These are your Atlas Airbnb notes. 

![Provide cluster name](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/provide_cluster_name_f5cd029131.png)

Step 3: Click “Next” and your sample_airbnb.listingsAndReviews will appear in the left-hand side of the console.

Step 4: Drag it directly under your Airbnb notes from your S3 bucket and hit “Save.” Your console should look like this when done:

![Saving S3 bucket](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/dragging_file_40f01a6f88.png)

Great job. You have successfully imported your Airbnb notes from both your S3 bucket and your Atlas cluster into one location. Let’s connect to our Federated Database and see our data combined in one easily query-able location. 

## Connect to your federated database
We are going to connect to our Federated Database using MongoDB Compass. 

Step 1: Click the green “Connect” button and then select “Connect using MongoDB Compass.” 

![Connect to FederationDemo](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/connect_to_federation_demo_769733e967.png)

Step 2: Copy in the connection string, making sure to switch out the user and password for your own. This user must have admin access in order to access the data. 

Step 3: Once you’re connected to Compass, click on “VirtualDatabase0” and once more on “VirtualCollection0.” 

![virtual database](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/virtual_database_c41050af02.png)

Amazing job. You can now look at all your Airbnb notes in one location! 

## Conclusion

In this tutorial, we have successfully stored your Airbnb data in various storage locations, combined these separate data sets into one via Data Federation, and successfully accessed our data back through MongoDB Compass. Now you can look for and book the perfect Airbnb for your trip in a fraction of the time. 

This tutorial will guide you through exactly how to combine your Airbnb data together for easier query-ability. 

MongoDB Data Federation Setup

<div>
    <img style="float: right;
        border-radius: 10px;
        margin-bottom: 30px;
        vertical-align: bottom;
        width: 30%;"
        src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/qs-badges/qs-badge-bson.png"
        alt="BSON Quickstart badge"
    />

Dates and times in programming can be a challenge. Which Time Zone is the event happening in? What date format is being used? Is it `MM/DD/YYYY` or `DD/MM/YYYY`? Settling on a standard is important for data storage and then again when displaying the date and time. The recommended way to store dates in MongoDB is to use the BSON Date data type.
</div>

The [BSON Specification](http://bsonspec.org/spec.html) refers to the `Date` type as the *UTC datetime* and is a 64-bit integer. It represents the number of milliseconds since the [Unix epoch](https://en.wikipedia.org/wiki/Unix_time), which was 00:00:00 UTC on 1 January 1970. This provides a lot of flexibilty in past and future dates. With a 64-bit integer in use, we are able to represent dates *roughly* 290 million years before and after the epoch. As a signed 64-bit integer we are able to represent dates *prior* to 1 Jan 1970 with a negative number and positive numbers represent dates *after* 1 Jan 1970.

## Why & Where to Use

You'll want to use the `Date` data type whenever you need to store date and/or time values in MongoDB. You may have seen a `timestamp` data type as well and thought "Oh, that's what I need." However, the `timestamp` data type should be left for **internal** usage in MongoDB. The `Date` type is the data type we'll want to use for application development.

## How to Use

There are some benefits to using the `Date` data type in that it comes with some handy features and methods. Need to assign a `Date` type to a variable? We have you covered there:

``` javascript
var newDate = new Date();
```

What did that create exactly?

``` none
> newDate;
ISODate("2020-05-11T20:14:14.796Z")
```

Very nice, we have a date and time wrapped as an ISODate. If we need that printed in a `string` format, we can use the `toString()` method.

``` none
> newDate.toString();
Mon May 11 2020 13:14:14 GMT-0700 (Pacific Daylight Time)
```

## Wrap Up

>Get started exploring BSON types, like Date, with [MongoDB Atlas](https://www.mongodb.com/cloud/atlas) today!

The `date` field is the recommended data type to use when you want to store date and time information in MongoDB. It provides the flexibility to store date and time values in a consistent format that can easily be stored and retrieved by your application. Give the BSON `Date` data type a try for your applications.

Working with dates and times can be a challenge. The Date BSON data type is an unsigned 64-bit integer with a UTC (Universal Time Coordinates) time zone.

Quick Start: BSON Data Types - Date

Learn how to perform CRUD operations using C Sharp for MongoDB databases.

MongoDB & C Sharp: CRUD Operations Tutorial

Daniel Coupal



<div class="introduction">

In this edition of the *Building with Patterns* series, we're going to
cover the Bucket Pattern. This pattern is particularly effective when
working with Internet of Things (IoT), Real-Time Analytics, or
Time-Series data in general. By *bucketing* data together we make it
easier to organize specific groups of data, increasing the ability to
discover historical trends or provide future forecasting and optimize
our use of storage.

</div>

<div class="content">

## The Bucket Pattern

With data coming in as a stream over a period of time (time series data)
we may be inclined to store each measurement in its own document.
However, this inclination is a very relational approach to handling the
data. If we have a sensor taking the temperature and saving it to the
database every minute, our data stream might look something like:

``` javascript
{
   sensor_id: 12345,
   timestamp: ISODate("2019-01-31T10:00:00.000Z"),
   temperature: 40
}

{
   sensor_id: 12345,
   timestamp: ISODate("2019-01-31T10:01:00.000Z"),
   temperature: 40
}

{
   sensor_id: 12345,
   timestamp: ISODate("2019-01-31T10:02:00.000Z"),
   temperature: 41
}
```

This can pose some issues as our application scales in terms of data and
index size. For example, we could end up having to index `sensor_id` and
`timestamp` for every single measurement to enable rapid access at the
cost of RAM. By leveraging the document data model though, we can
"bucket" this data, by time, into documents that hold the measurements
from a particular time span. We can also programmatically add additional
information to each of these "buckets".

By applying the Bucket Pattern to our data model, we get some benefits
in terms of index size savings, potential query simplification, and the
ability to use that pre-aggregated data in our documents. Taking the
data stream from above and applying the Bucket Pattern to it, we would
wind up with:

``` javascript
{
    sensor_id: 12345,
    start_date: ISODate("2019-01-31T10:00:00.000Z"),
    end_date: ISODate("2019-01-31T10:59:59.000Z"),
    measurements: [
       {
       timestamp: ISODate("2019-01-31T10:00:00.000Z"),
       temperature: 40
       },
       {
       timestamp: ISODate("2019-01-31T10:01:00.000Z"),
       temperature: 40
       },
       ...
       {
       timestamp: ISODate("2019-01-31T10:42:00.000Z"),
       temperature: 42
       }
    ],
   transaction_count: 42,
   sum_temperature: 2413
}
```

By using the Bucket Pattern, we have "bucketed" our data to, in this
case, a one hour bucket. This particular data stream would still be
growing as it currently only has 42 measurements; there's still more
measurements for that hour to be added to the "bucket". When they are
added to the `measurements` array, the `transaction_count` will be
incremented and `sum_temperature` will also be updated.

With the pre-aggregated `sum_temperature` value, it then becomes
possible to easily pull up a particular bucket and determine the average
temperature (`sum_temperature / transaction_count`) for that bucket.
When working with time-series data it is frequently more interesting and
important to know what the average temperature was from 2:00 to 3:00 pm
in Corning, California on 13 July 2018 than knowing what the temperature
was at 2:03 pm. By bucketing and doing pre-aggregation we're more able
to easily provide that information.

Additionally, as we gather more and more information we may determine
that keeping all of the source data in an archive is more effective. How
frequently do we need to access the temperature for Corning from 1948,
for example? Being able to move those buckets of data to a data archive
can be a large benefit.

## Sample Use Case

One example of making time-series data valuable in the real world comes
from an [IoT implementation by
Bosch](https://www.mongodb.com/customers/bosch). They are using MongoDB
and time-series data in an automotive field data app. The app captures
data from a variety of sensors throughout the vehicle allowing for
improved diagnostics of the vehicle itself and component performance.

Other examples include major banks that have incorporated this pattern
in financial applications to group transactions together.

</div>

<div class="summary">

## Conclusion

When working with time-series data, using the Bucket Pattern in MongoDB
is a great option. It reduces the overall number of documents in a
collection, improves index performance, and by leveraging
pre-aggregation, it can simplify data access.

The Bucket Design pattern works great for many cases. But what if there
are outliers in our data? That's where the next pattern we'll discuss,
the [Outlier Design
Pattern](/how-to/outlier-pattern), comes into play.

</div>


Over the course of this blog post series, we'll take a look at twelve common Schema Design Patterns that work well in MongoDB.

Building with Patterns: The Bucket Pattern

<div>
    <img style="float: right;
        border-radius: 10px;
        margin-bottom: 30px;
        vertical-align: bottom;
        width: 30%;"
        src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/qs-badges/qs-badge-bson.png"
        alt="BSON Quickstart badge"
    />

In the database world, it is frequently important to have unique identifiers associated with a record. In a legacy, tabular database, these unique identifiers are often used as primary keys. In a modern database, such as MongoDB, we need a unique identifier in an `_id` field as a primary key as well. MongoDB provides an automatic unique identifier for the `_id` field in the form of an `ObjectId` data type.

</div>

For those that are familiar with MongoDB [Documents](https://alger.me/mongodb-documents) you've likely come across the `ObjectId` data type in the `_id` field. For those unfamiliar with MongoDB Documents, the [ObjectId](https://alger.me/mongodb-objectid) datatype is automatically generated as a unique document identifier if no other identifier is provided. But what is an `ObjectId` field? What makes them unique? This post will unveil some of the magic behind the BSON ObjectId data type.  First, though, what is BSON?

## Binary JSON (BSON)

Many programming languages have JavaScript Object Notation (JSON) support or similar data structures. MongoDB uses JSON documents to store records. However, behind the scenes, MongoDB represents these documents in a binary-encoded format called [BSON](http://bsonspec.org/). BSON provides additional data types and ordered fields to allow for efficient support across a variety of languages. One of these additional data types is ObjectId.

## Makeup of an ObjectId

Let's start with an examination of what goes into an ObjectId. If we take a look at the construction of the ObjectId value, in its current implementation, it is a 12-byte hexadecimal value. This 12-byte configuration is smaller than a typical [universally unique identifier](https://alger.me/uuid-defined) (UUID), which is, typically, 128-bits. Beginning in MongoDB 3.4, an ObjectId consists of the following values:

-   4-byte value representing the seconds since the [Unix epoch](https://en.wikipedia.org/wiki/Unix_time),
-   5-byte random value, and
-   3-byte counter, starting with a random value.

![BSON ObjectId diagram](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/objectid.png)

With this makeup, ObjectIds are *likely* to be globally unique and unique per collection. Therefore, they make a good candidate for the unique requirement of the `_id` field. While the `_id` in a collection can be an auto-assigned `ObjectId`, it can be user-defined as well, as long as it is unique within a collection. Remember that if you aren't using a MongoDB generated `ObjectId` for the `_id` field, the application creating the document will have to ensure the value is unique.

## History of ObjectId

The makeup of the ObjectId has changed over time. Through version 3.2, it consisted of the following values:

-   4-byte value representing the seconds since the Unix epoch,
-   3-byte machine identifier,
-   2-byte process id, and
-   3-byte counter, starting with a random value.

The change from including a machine-specific identifier and process id to a random value increased the likelihood that the `ObjectId` would be globally unique. These machine-specific 5-bytes of information became less likely to be random with the prevalence of Virtual Machines (VMs) that had the same MAC addresses and processes that started in the same order. While it still isn't guaranteed, removing machine-specific information from the `ObjectId` increases the chances that the same machine won't generate the same `ObjectId`.

## ObjectId Odds of Uniqueness

The randomness of the last eight bytes in the current implementation makes the likelihood of the same ObjectId being created pretty small.  How small depends on the number of inserts per second that your application does. Let's do some quick math and look at the odds.

If we do one insert per second, the first four bytes of the ObjectId would change so we can't have a duplicate ObjectId. What are the odds though when multiple documents are inserted in the same second that *two* ObjectIds are the same? Since there are *eight* bits in a byte, and *eight* random bytes in our Object Id (5 random + 3 random starting values), the denominator in our odds ratio would be 2^(8\*8), or 1.84467441x10'^19. For those that have forgotten scientific notation, that's 18,446,744,100,000,000,000. Yes, that's correct, 18 quintillion and change. As a bit of perspective, the odds of being struck by lightning in the U.S. in a given year are 1 in 700,000, according to [National Geographic](https://www.nationalgeographic.com/news/2005/6/flash-facts-about-lightning/).  The odds of winning the [Powerball Lottery](https://www.powerball.com/games/home) jackpot are 1 in 292,201,338. The numerator in our odds equation is the number of documents per second. Even in a write-heavy system with 250 million writes/second, the odds are, while not zero, pretty good against duplicate ObjectIds being generated.

## Wrap Up

>Get started exploring BSON types, like ObjectId, with [MongoDB Atlas](https://www.mongodb.com/cloud/atlas) today!


ObjectId is one data type that is part of the [BSON Specification](http://bsonspec.org/) that MongoDB uses for data storage. It is a binary representation of JSON and includes [other data types](https://www.mongodb.com/blog/post/the-top-12-bson-data-types-you-wont-find-in-json) beyond those defined in JSON. It is a powerful data type that is incredibly useful as a unique identifier in MongoDB Documents.

MongoDB provides an automatic unique identifier for the _id field in the form of an ObjectId data type.

Quick Start: BSON Data Types - ObjectId

<div>
    <img style="float: right;
        border-radius: 10px;
        margin-bottom: 30px;
        vertical-align: bottom;
        width: 30%;"
        src="https://devhub-cms.corp.mongodb.com/public/new-articles/qs-badges/qs-badge-bson.png"
        alt="BSON Quickstart badge"
    />

Think back to when you were first introduced to the concept of decimals in numerical calculations. Doing math problems along the lines of 3.231 / 1.28 caused problems when starting out because 1.28 doesn't go into 3.231 evenly. This causes a long string of numbers to be created to provide a more precise answer. In programming languages, we must choose which number format is correct depending on the amount of precision we need. When one needs high precision when working with [BSON data types](http://bsonspec.org/), the `decimal128` is the one to use.
</div>

As the name suggests, decimal128 provides 128 bits of decimal representation for storing really big (or really small) numbers when rounding decimals exactly is important. Decimal128 supports 34 decimal digits of precision, or [significand](https://en.wikipedia.org/wiki/Significand) along with an exponent range of -6143 to +6144. The significand is not normalized in the decimal128 standard allowing for multiple possible representations: 10 x 10^-1 = 1 x 10^0 = .1 x 10^1 = .01 x 10^2 and so on. Having the ability to store maximum and minimum values in the order of 10^6144 and 10^-6143, respectively, allows for a lot of precision.

![Decimal128 Precision Details](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/decimal128-precision-details.png)

## Why & Where to Use

Sometimes when doing mathematical calculations in a programmatic way, results are unexpected. For example in Node.js:

``` bash
> 0.1
0.1
> 0.2
0.2
> 0.1 * 0.2
0.020000000000000004
> 0.1 + 0.1
0.010000000000000002
```

This issue is not unique to Node.js, in Java:

``` java
class Main {
 public static void main(String[] args) {
   System.out.println("0.1 * 0.2:");
   System.out.println(0.1 * 0.2);
 }
}
```

Produces an output of:

``` bash
0.1 * 0.2:
0.020000000000000004
```

The same computations in Python, Ruby, Rust, and others produce the same results. What's going on here? Are these languages just bad at math? Not really, binary floating-point numbers just aren't great at representing base 10 values. For example, the `0.1` used in the above examples is represented in binary as `0.0001100110011001101`.

For many situations, this isn't a huge issue. However, in monetary applications precision is very important. Who remembers the [half-cent issue from Superman III](https://www.youtube.com/watch?v=iLw9OBV7HYA)?  When precision and accuracy are important for computations, decimal128 should be the data type of choice.

## How to Use

In MongoDB, storing data in decimal128 format is relatively straight forward with the [NumberDecimal()](https://docs.mongodb.com/manual/core/shell-types/#shell-type-decimal) constructor:

``` bash
NumberDecimal("9823.1297")
```

Passing in the decimal value as a string, the value gets stored in the database as:

``` bash
NumberDecimal("9823.1297")
```

If values are passed in as `double` values:

``` bash
NumberDecimal(1234.99999999999)
```

Loss of precision can occur in the database:

``` bash
NumberDecimal("1234.50000000000")
```

Another consideration, beyond simply the usage in MongoDB, is the usage and support your programming has for decimal128. Many languages don't natively support this feature and will require a plugin or additional package to get the functionality. Some examples...

Python: The [`decimal.Decimal`](https://docs.python.org/3/library/decimal.html) module can be used for floating-point arithmetic.

Java: The [Java BigDecimal](https://docs.oracle.com/javase/1.5.0/docs/api/java/math/BigDecimal.html) class provides support for decimal128 numbers.

Node.js: There are several packages that provide support, such as [js-big-decimal](https://www.npmjs.com/package/js-big-decimal) or [node.js bigdecimal](https://www.npmjs.com/package/bigdecimal) available on [npm](https://www.npmjs.com).

## Wrap Up

>Get started exploring BSON types, like decimal128, with [MongoDB Atlas](https://www.mongodb.com/cloud/atlas) today!

The `decimal128` field came about in August 2009 as part of the [IEEE 754-2008](https://en.wikipedia.org/wiki/IEEE_754-2008_revision) revision of floating points. MongoDB 3.4 is when support for decimal128 first appeared and to use the `decimal` data type with MongoDB, you'll want to make sure you [use a driver](https://docs.mongodb.com/ecosystem/drivers/) version that supports this great feature. Decimal128 is great for huge (or very tiny) numbers and for when precision in those numbers is important.

Working with decimal numbers can be a challenge. The Decimal128 BSON data type allows for high precision options when working with numbers.

Quick Start: BSON Data Types - Decimal128

<div>
    <img style="float: right;
        border-radius: 10px;
        margin-bottom: 30px;
        vertical-align: bottom;
        width: 30%;"
        src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/qs-badges/qs-badge-aggregation-framework.png"
        alt="BSON Quickstart badge"
    />

One of the difficulties when storing any data is knowing how it will be accessed in the future. What reports need to be run on it? What information is "hidden" in there that will allow for meaningful insights for your business? After spending the time to [design your data schema](https://www.mongodb.com/blog/post/building-with-patterns-a-summary) in an appropriate fashion for your application, one needs to be able to retrieve it. In MongoDB, there are two basic ways that data retrieval can be done: through queries with the [find()](https://docs.mongodb.com/manual/reference/method/db.collection.find/) command, and through analytics using the aggregation framework and the [aggregate()](https://docs.mongodb.com/manual/reference/method/db.collection.aggregate/) command.

</div>

`find()` allows for the querying of data based on a condition. One can filter results, do basic document transformations, sort the documents, limit the document result set, etc. The `aggregate()` command opens the door to a whole new world with the [aggregation framework](https://docs.mongodb.com/manual/aggregation/). In this series of posts, I'll take a look at some of the reasons why using the aggregation framework is so powerful, and how to harness that power.

## Why Aggregate with MongoDB?

A frequently asked question is why do aggregation inside MongoDB at all?  From the MongoDB documentation:

>
>
>Aggregation operations process data records and return computed results.  Aggregation operations group values from multiple documents together, and can perform a variety of operations on the grouped data to return a single result.
>
>

By using the built-in aggregation operators available in MongoDB, we are able to do analytics on a cluster of servers we're already using without having to move the data to another platform, like Apache [Spark](https://spark.apache.org/) or [Hadoop](https://hadoop.apache.org/). While those, and similar, platforms are fast, the data transfer from MongoDB to them can be slow and potentially expensive. By using the aggregation framework the work is done inside MongoDB and then the final results can be sent to the application typically resulting in a smaller amount of data being moved around. It also allows for the querying of the **LIVE** version of the data and not an older copy of data from a batch.

Aggregation in MongoDB allows for the transforming of data and results in a more powerful fashion than from using the `find()` command. Through the use of multiple stages and expressions, you are able to build a "pipeline" of operations on your data to perform analytic operations.  What do I mean by a "pipeline"? The aggregation framework is conceptually similar to the `*nix` command line pipe, `|`. In the `*nix` command line pipeline, a pipe transfers the standard output to some other destination. The output of one command is sent to another command for further processing.

![*nix pipeline example](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/agg-framework/nix-pipeline.gif)

In the aggregation framework, we think of stages instead of commands.  And the stage "output" is documents. Documents go into a stage, some work is done, and documents come out. From there they can move onto another stage or provide output.

## Aggregation Stages

At the time of this writing, there are twenty-eight different aggregation stages available. These different stages provide the ability to do a wide variety of tasks. For example, we can build an aggregation pipeline that *matches* a set of documents based on a set of criteria, *groups* those documents together, *sorts* them, then returns that result set to us.

![Aggreggation Pipeline example](https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/agg-framework/agg-framework-pipeline.gif)

Or perhaps our pipeline is more complicated and the document flows through the `$match`, `$unwind`, `$group`, `$sort`, `$limit`, `$project`, and finally a `$skip` stage.

This can be confusing and some of these concepts are worth repeating.  Therefore, let's break this down a bit further:

-   A pipeline starts with documents
-   These documents come from a collection, a view, or a specially designed stage
-   In each stage, documents enter, work is done, and documents exit
-   The stages themselves are defined using the document syntax

Let's take a look at an example pipeline. Our documents are from the [Sample Data](https://docs.atlas.mongodb.com/sample-data/available-sample-datasets/) that's available in MongoDB Atlas and the `routes` collection in the `sample_training` database. Here's a sample document:

``` json
{
"_id":{
    "$oid":"56e9b39b732b6122f877fa31"
},
"airline":{
   "id":{
       "$numberInt":"410"
   },
   "name":"Aerocondor"
   ,"alias":"2B"
   ,"iata":"ARD"
},
"src_airport":"CEK",
"dst_airport":"KZN",
"Codeshare":"",
"stops":{
    "$numberInt":"0"
},
"airplane":"CR2"
}
```

>
>
>If you haven't yet set up your free cluster on [MongoDB Atlas](http://bit.ly/mongodbatlas), now is a great time to do so. You have all the instructions in this [blog post](/quickstart/free-atlas-cluster).
>
>

For this example query, let's find the top three airlines that offer the most direct flights out of the airport in Portland, Oregon, USA (PDX).  To start with, we'll do a `$match` stage so that we can concentrate on doing work only on those documents that meet a base of conditions. In this case, we'll look for documents with a `src_airport`, or source airport, of PDX and that are direct flights, i.e. that have zero stops.

``` javascript
{
  $match: {
    "src_airport": "PDX",
    "stops": 0
  }
}
```

That reduces the number of documents in our pipeline down from 66,985 to 113. Next, we'll group by the airline name and count the number of flights:

``` javascript
{
    $group: {
        _id: {
            "airline name": "$airline.name"
        },
        count: {
            $sum: 1
        }
    }
}
```

With the addition of the `$group` stage, we're down to 16 documents.  Let's sort those with a `$sort` stage and sort in descending order:

``` javascript
{
    $sort: {
        count: -1
}
```

Then we can add a `$limit` stage to just have the top three airlines that are servicing Portland, Oregon:

``` javascript
{
   $limit: 3
}
```

After putting the documents in the `sample_training.routes` collection through this aggregation pipeline, our results show us that the top three airlines offering non-stop flights departing from PDX are Alaska, American, and United Airlines with 39, 17, and 13 flights, respectively.

How does this look in code? It's fairly straightforward with using the `db.aggregate()` function. For example, in Python you would do something like:

``` python
from pymongo import MongoClient

# Requires the PyMongo package.
# The dnspython package is also required to use a mongodb+src URI string
# https://api.mongodb.com/python/current

client = MongoClient('YOUR-ATLAS-CONNECTION-STRING')
result = client['sample_training']['routes'].aggregate([
    {
        '$match': {
            'src_airport': 'PDX',
            'stops': 0
        }
    }, {
        '$group': {
            '_id': {
                'airline name': '$airline.name'
            },
            'count': {
                '$sum': 1
            }
        }
    }, {
        '$sort': {
            'count': -1
        }
    }, {
        '$limit': 3
    }
])
```

The aggregation code is pretty similar in other languages as well.

## Wrap Up

The MongoDB aggregation framework is an extremely powerful set of tools.  The processing is done on the server itself which results in less data being sent over the network. In the example used here, instead of pulling **all** of the documents into an application and processing them in the application, the aggregation framework allows for only the three documents we wanted from our query to be sent back to the application.

This was just a brief introduction to some of the operators available.  Over the course of this series, I'll take a closer look at some of the most popular aggregation framework operators as well as some interesting, but less used ones. I'll also take a look at performance considerations of using the aggregation framework.


Learn about MongoDB's aggregation framework and aggregation operators.

Introduction to the MongoDB Aggregation Framework



<div class="introduction">

Welcome back to the Building with Patterns series. Last time we looked
at the [Polymorphic Pattern](https://www.mongodb.com/developer/products/mongodb/polymorphic-pattern) which covers
situations when all documents in a collection are of similar, but not
identical, structure. In this post, we'll take a look at the Attribute
Pattern.

The Attribute Pattern is particularly well suited when:

-   We have big documents with many similar fields but there is a subset of fields that share common characteristics and we want to sort or query on that subset of fields, *or*
-   The fields we need to sort on are only found in a small subset of  documents, *or*
-   Both of the above conditions are met within the documents.

For performance reasons, to optimize our search we'd likely need many indexes to account for all of the subsets. Creating all of these indexes could reduce performance. The Attribute Pattern provides a good solution for these cases.

## The Attribute Pattern

Let's think about a collection of movies. The documents will likely have similar fields involved across all of the documents: title, director,
producer, cast, etc. Let's say we want to search on the release date. A
challenge that we face when doing so, is *which* release date? Movies
are often released on different dates in different countries.

``` javascript
{
    title: "Star Wars",
    director: "George Lucas",
    ...
    release_US: ISODate("1977-05-20T01:00:00+01:00"),
    release_France: ISODate("1977-10-19T01:00:00+01:00"),
    release_Italy: ISODate("1977-10-20T01:00:00+01:00"),
    release_UK: ISODate("1977-12-27T01:00:00+01:00"),
    ...
}
```

A search for a release date will require looking across many fields at
once. In order to quickly do searches for release dates, we'd need
several indexes on our movies collection:

``` javascript
{release_US: 1}
{release_France: 1}
{release_Italy: 1}
...
```

By using the Attribute Pattern, we can move this subset of information into an array and reduce the indexing needs. We turn this information into an array of key-value pairs:

``` javascript
{
    title: "Star Wars",
    director: "George Lucas",
    ...
    releases: [
        {
        location: "USA",
        date: ISODate("1977-05-20T01:00:00+01:00")
        },
        {
        location: "France",
        date: ISODate("1977-10-19T01:00:00+01:00")
        },
        {
        location: "Italy",
        date: ISODate("1977-10-20T01:00:00+01:00")
        },
        {
        location: "UK",
        date: ISODate("1977-12-27T01:00:00+01:00")
        },
        ...
    ],
    ...
}
```

Indexing becomes much more manageable by creating one index on the
elements in the array:

``` javascript
{ "releases.location": 1, "releases.date": 1}
```

By using the Attribute Pattern, we can add organization to our documents for common characteristics and account for rare/unpredictable fields. For example, a movie released in a new or small festival. Further, moving to a key/value convention allows for the use of non-deterministic naming and the easy addition of qualifiers. For example, if our data collection was on bottles of water, our attributes might look something like:

``` javascript
"specs": [
    { k: "volume", v: "500", u: "ml" },
    { k: "volume", v: "12", u: "ounces" }
]
```

Here we break the information out into keys and values, "k" and "v," and add in a third field, "u," which allows for the units of measure to be stored separately.

``` javascript
{"specs.k": 1, "specs.v": 1, "specs.u": 1}
```

## Sample use case

The Attribute Pattern is well suited for schemas that have sets of fields that have the same value type, such as lists of dates. It also works well when working with the characteristics of products. Some products, such as clothing, may have sizes that are expressed in small, medium, or large. Other products in the same collection may be expressed in volume. Yet others may be expressed in physical dimensions or weight.

A customer in the domain of asset management recently deployed their solution using the Attribute Pattern. The customer uses the pattern to store all characteristics of a given asset. These characteristics are seldom common across the assets or are simply difficult to predict at design time. Relational models typically use a complicated design process to express the same idea in the form of [user-defined fields](https://martinfowler.com/bliki/UserDefinedField.html).

While many of the fields in the product catalog are similar, such as name, vendor, manufacturer, country of origin, etc., the specifications, or attributes, of the item may differ. If your application and data access patterns rely on searching through many of these different fields at once, the Attribute Pattern provides a good structure for the data.

## Conclusion

The Attribute Pattern provides for easier indexing the documents, targeting many similar fields per document. By moving this subset of data into a key-value sub-document, we can use non-deterministic field names, add additional qualifiers to the information, and more clearly state the relationship of the original field and value. When we use the Attribute Pattern, we need fewer indexes, our queries become simpler to write, and our queries become faster.


Building with Patterns: The Attribute Pattern



<div class="introduction">

We've looked at various ways of optimally storing data in the **Building
with Patterns** series. Now, we're going to look at a different aspect
of schema design. Just storing data and having it available isn't,
typically, all that useful. The usefulness of data becomes much more
apparent when we can compute values from it. What's the total sales
revenue of the latest Amazon Alexa? How many viewers watched the latest
blockbuster movie? These types of questions can be answered from data
stored in a database but must be computed.

Running these computations every time they're requested though becomes a
highly resource-intensive process, especially on huge datasets. CPU
cycles, disk access, memory all can be involved.

Think of a movie information web application. Every time we visit the
application to look up a movie, the page provides information about the
number of cinemas the movie has played in, the total number of people
who've watched the movie, and the overall revenue. If the application
has to constantly compute those values for each page visit, it could use
a lot of processing resources on popular movies

Most of the time, however, we don't need to know those exact numbers. We
could do the calculations in the background and update the main movie
information document once in a while. These **computations** then allow
us to show a valid representation of the data without having to put
extra effort on the CPU.

</div>

<div class="content">

## The Computed Pattern

The Computed Pattern is utilized when we have data that needs to be
computed repeatedly in our application. The Computed Pattern is also
utilized when the data access pattern is read intensive; for example, if
you have 1,000,000 reads per hour but only 1,000 writes per hour, doing
the computation at the time of a write would divide the number of
calculations by a factor 1000.

In our movie database example, we can do the computations based on all
of the screening information we have on a particular movie, compute the
result(s), and store them with the information about the movie itself.
In a low write environment, the computation could be done in conjunction
with any update of the source data. Where there are more regular writes,
the computations could be done at defined intervals - every hour for
example. Since we aren't interfering with the source data in the
screening information, we can continue to rerun existing calculations or
run new calculations at any point in time and know we will get correct
results.

Other strategies for performing the computation could involve, for
example, adding a timestamp to the document to indicate when it was last
updated. The application can then determine when the computation needs
to occur. Another option might be to have a queue of computations that
need to be done. Selecting the update strategy is best left to the
application developer.

## Sample Use Case

The **Computed Pattern** can be utilized wherever calculations need to
be run against data. Datasets that need sums, such as revenue or
viewers, are a good example, but time series data, product catalogs,
single view applications, and event sourcing are prime candidates for
this pattern too.

This is a pattern that many customers have implemented. For example, a
customer does massive aggregation queries on vehicle data and store the
results for the server to show the info for the next few hours.

A publishing company compiles all kind of data to create ordered lists
like the "100 Best...". Those lists only need to be regenerated once in
a while, while the underlying data may be updated at other times.

</div>

<div class="summary">

## Conclusion

This powerful design pattern allows for a reduction in CPU workload and
increased application performance. It can be utilized to apply a
computation or operation on data in a collection and store the result in
a document. This allows for the avoidance of the same computation being
done repeatedly. Whenever your system is performing the same
calculations repeatedly and you have a high read to write ratio,
consider the **Computed Pattern**.

We're over a third of the way through this **Building with Patterns**
series. Next time we'll look at the features and benefits of the [Subset
Pattern](https://www.mongodb.com/blog/post/building-with-patterns-the-subset-pattern)
and how it can help with memory shortage issues.

</div>


Building with Patterns: The Computed Pattern

Large objects, or "files", are easily stored in MongoDB. It is no problem to store 100MB videos in the database.

This has a number of advantages over files stored in a file system. Unlike a file system, the database will have no problem dealing with millions of objects. Additionally, we get the power of the database when dealing with this data: we can do advanced queries to find a file, using indexes; we can also do neat things like replication of the entire file set.

MongoDB stores objects in a binary format called [BSON](http://bsonspec.org). BinData is a BSON data type for a binary byte array. However, MongoDB objects are typically limited to 16MB in size. To deal with this, files are "chunked" into multiple objects that are less than 255 KiB each. This has the added advantage of letting us efficiently retrieve a specific range of the given file.

While we could write our own chunking code, a standard format for this chunking is predefined, called [GridFS](http://docs.mongodb.com/manual/core/gridfs/). GridFS support is included in all official MongoDB drivers and also in the [mongofiles](http://docs.mongodb.com/manual/reference/program/mongofiles/) command line utility.

A good way to do a quick test of this facility is to try out the mongofiles utility. See the [MongoDB documentation](http://docs.mongodb.com/manual/core/gridfs/) for more information on GridFS.

## More Information

-   [GridFS Docs](http://docs.mongodb.com/manual/core/gridfs/)
-   [Building MongoDB Applications with Binary Files Using GridFS: Part 1](http://www.mongodb.com/blog/post/building-mongodb-applications-binary-files-using-gridfs-part-1)
-   [Building MongoDB Applications with Binary Files Using GridFS: Part 2](http://www.mongodb.com/blog/post/building-mongodb-applications-binary-files-using-gridfs-part-2)
-   [MongoDB Architecture Guide](https://www.mongodb.com/lp/white-paper/architecture-guide)

Discover how to store large objects and files in MongoDB.

Storing Large Objects and Files in MongoDB



<div class="introduction">

Some years ago, the first PCs had a whopping 256KB of RAM and dual 5.25"
floppy drives. No hard drives as they were incredibly expensive at the
time. These limitations resulted in having to physically swap floppy
disks due to a lack of memory when working with large (for the time)
amounts of data. If only there was a way back then to only bring into
memory the data I frequently used, as in a subset of the overall data.

Modern applications aren't immune from exhausting resources. MongoDB
keeps frequently accessed data, referred to as the [working set](https://docs.mongodb.com/manual/reference/glossary/#term-working-set),
in RAM. When the working set of data and indexes grows beyond the
physical RAM allotted, performance is reduced as disk accesses starts to
occur and data rolls out of RAM.

How can we solve this? First, we could add more RAM to the server. That
only scales so much though. We can look at
[sharding](https://docs.mongodb.com/manual/sharding/)
our collection, but that comes with additional costs and complexities
that our application may not be ready for. Another option is to reduce
the size of our working set. This is where we can leverage the Subset
Pattern.

</div>

<div class="content">

## The Subset Pattern

This pattern addresses the issues associated with a working set that
exceeds RAM, resulting in information being removed from memory. This is
frequently caused by large documents which have a lot of data that isn't
actually used by the application. What do I mean by that exactly?

Imagine an e-commerce site that has a list of reviews for a product.
When accessing that product's data it's quite possible that we'd only
need the most recent ten or so reviews. Pulling in the entirety of the
product data with **all** of the reviews could easily cause the working
set to expand.

<img src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/how-to/design-patterns/subset-full-doc.png" class="align-center" alt="A full document with reviews" />

Instead of storing all the reviews with the product, we can split the
collection into two collections. One collection would have the most
frequently used data, e.g. current reviews and the other collection
would have less frequently used data, e.g. old reviews, product history,
etc. We can duplicate part of a 1-N or N-N relationship that is used by
the most used side of the relationship.

<img src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/how-to/design-patterns/subset-two-collections.png" class="align-center" alt="A document with a subset and a full review collection" />

In the **Product** collection, we'll only keep the ten most recent
reviews. This allows the working set to be reduced by only bringing in a
portion, or subset, of the overall data. The additional information,
reviews in this example, are stored in a separate **Reviews** collection
that can be accessed if the user wants to see additional reviews. When
considering where to split your data, the most used part of the document
should go into the "main" collection and the less frequently used data
into another. For our reviews, that split might be the number of reviews
visible on the product page.

## Sample Use Case

The Subset Pattern is very useful when we have a large portion of data
inside a document that is rarely needed. Product reviews, article
comments, actors in a movie are all examples of use cases for this
pattern. Whenever the document size is putting pressure on the size of
the working set and causing the working set to exceed the computer's RAM
capacities, the Subset Pattern is an option to consider.

</div>

<div class="summary">

## Conclusion

By using smaller documents with more frequently accessed data, we reduce
the overall size of the working set. This allows for shorter disk access
times for the most frequently used information that an application
needs. One tradeoff that we must make when using the Subset Pattern is
that we must manage the subset and also if we need to pull in older
reviews or all of the information, it will require additional trips to
the database to do so.

The next post in this series will look at the features and benefits of
the [Extended Reference Pattern](https://www.mongodb.com/blog/post/building-with-patterns-the-extended-reference-pattern).

</div>


Building with Patterns: The Subset Pattern



<div class="introduction">

So far in this *Building with Patterns* series, we've looked at the
[Polymorphic](/how-to/polymorphic-pattern),
[Attribute](/how-to/attribute-pattern), and
[Bucket](/how-to/bucket-pattern) patterns. While the document schema in
these patterns has slight variations, from an application and query
standpoint, the document structures are fairly consistent. What happens,
however, when this isn't the case? What happens when there is data that
falls outside the "normal" pattern? What if there's an outlier?

Imagine you are starting an e-commerce site that sells books. One of the
queries you might be interested in running is "who has purchased a
particular book". This could be useful for a recommendation system to
show your customers similar books of interest. You decide to store the
`user_id` of a customer in an array for each book. Simple enough, right?

Well, this may indeed work for 99.99% of the cases, but what happens
when J.K. Rowling releases a new Harry Potter book and sales spike in
the millions? The 16MB [BSON document](https://www.mongodb.com/json-and-bson) size limit could easily
be reached. Redesigning our entire application for this *outlier*
situation could result in reduced performance for the typical book, but
we do need to take it into consideration.

</div>

<div class="content">

## The Outlier Pattern

With the Outlier Pattern, we are working to prevent a few queries or
documents driving our solution towards one that would not be optimal for
the majority of our use cases. Not every book sold will sell millions of
copies.

A typical `book` document storing `user_id` information might look
something like:

``` javascript
{
    "_id": ObjectID("507f1f77bcf86cd799439011")
    "title": "A Genealogical Record of a Line of Alger",
    "author": "Ken W. Alger",
    ...,
    "customers_purchased": ["user00", "user01", "user02"]

}
```

This would work well for a large majority of books that aren't likely to
reach the "best seller" lists. Accounting for outliers though results in
the `customers_purchased` array expanding beyond a 1000 item limit we
have set, we'll add a new field to "flag" the book as an outlier.

``` javascript
{
    "_id": ObjectID("507f191e810c19729de860ea"),
    "title": "Harry Potter, the Next Chapter",
    "author": "J.K. Rowling",
    ...,
   "customers_purchased": ["user00", "user01", "user02", ..., "user999"],
   "has_extras": "true"
}
```

We'd then move the overflow information into a separate document linked
with the book's `id`. Inside the application, we would be able to
determine if a document has a `has_extras` field with a value of `true`.
If that is the case, the application would retrieve the extra
information. This could be handled so that it is rather transparent for
most of the application code.

Many design decisions will be based on the application workload, so this
solution is intended to show an example of the Outlier Pattern. The
important concept to grasp here is that the outliers have a substantial
enough difference in their data that, if they were considered "normal",
changing the application design for them would degrade performance for
the more typical queries and documents.

## Sample Use Case

The Outlier Pattern is an advanced pattern, but one that can result in
large performance improvements. It is frequently used in situations when
popularity is a factor, such as in social network relationships, book
sales, movie reviews, etc. The Internet has transformed our world into a
much smaller place and when something becomes popular, it transforms the
way we need to model the data around the item.

One example is a customer that has a video conferencing product. The
list of authorized attendees in most video conferences can be kept in
the same document as the conference. However, there are a few events,
like a company's all hands, that have thousands of expected attendees.
For those outlier conferences, the customer implemented "overflow"
documents to record those long lists of attendees.

</div>

<div class="summary">

## Conclusion

The problem that the Outlier Pattern addresses is preventing a few
documents or queries to determine an application's solution. Especially
when that solution would not be optimal for the majority of use cases.
We can leverage MongoDB's flexible data model to add a field to the
document "flagging" it as an outlier. Then, inside the application, we
handle the outliers slightly differently. By tailoring your schema for
the typical document or query, application performance will be optimized
for those normal use cases and the outliers will still be addressed.

One thing to consider with this pattern is that it often is tailored for
specific queries and situations. Therefore, ad hoc queries may result in
less than optimal performance. Additionally, as much of the work is done
within the application code itself, additional code maintenance may be
required over time.

In our next *Building with Patterns* post, we'll take a look at the
[Computed Pattern](/how-to/computed-pattern) and how to optimize schema for
applications that can result in unnecessary waste of resources.
 
</div>


Building with Patterns: The Outlier Pattern



<div class="introduction">

## Introduction

One frequently asked question when it comes to MongoDB is "How do I
structure my schema in MongoDB for my application?" The honest answer
is, it depends. Does your application do more reads than writes? What
data needs to be together when read from the database? What performance
considerations are there? How large are the documents? How large will
they get? How do you anticipate your data will grow and scale?

All of these questions, and more, factor into how one designs a database
schema in MongoDB. It has been said that MongoDB is schemaless. In fact,
schema design is very important in MongoDB. The hard fact is that most
performance issues we've found trace back to poor schema design.

Over the course of this series, Building with Patterns, we'll take a
look at twelve common Schema Design Patterns that work well in MongoDB.
We hope this series will establish a common methodology and vocabulary
you can use when designing schemas. Leveraging these patterns allows for
the use of "building blocks" in schema planning, resulting in more
methodology being used than art.

MongoDB uses a [document data
model](https://docs.mongodb.com/manual/core/data-model-design/). This
model is inherently flexible, allowing for data models to support your
application needs. The flexibility also can lead to schemas being more
complex than they should. When thinking of schema design, we should be
thinking of performance, scalability, and simplicity.

Let's start our exploration into schema design with a look at what can
be thought as the base for all patterns, the *Polymorphic Pattern*. This
pattern is utilized when we have documents that have more similarities
than differences. It's also a good fit for when we want to keep
documents in a single collection.

</div>

<div class="content">

## The Polymorphic Pattern

When all documents in a collection are of similar, but not identical,
structure, we call this the Polymorphic Pattern. As mentioned, the
Polymorphic Pattern is useful when we want to access (query) information
from a single collection. Grouping documents together based on the
queries we want to run (instead of separating the object across tables
or collections) helps improve performance.

Imagine that our application tracks professional sports athletes across
all different sports.

We still want to be able to access all of the athletes in our
application, but the attributes of each athlete are very different. This
is where the Polymorphic Pattern shines. In the example below, we store
data for athletes from two different sports in the same collection. The
data stored about each athlete does not need to be the same even though
the documents are in the same collection.

<img src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/Polymorphic1-nanehssyv3.png" width="760" alt="Polymorphic Design Pattern with Common Fields" />

Professional athlete records have some similarities, but also some
differences. With the Polymorphic Pattern, we are easily able to
accommodate these differences. If we were not using the Polymorphic
Pattern, we might have a collection for Bowling Athletes and a
collection for Tennis Athletes. When we wanted to query on all athletes,
we would need to do a time-consuming and potentially complex join.
Instead, since we are using the Polymorphic Pattern, all of our data is
stored in one Athletes collection and querying for all athletes can be
accomplished with a simple query.

This design pattern can flow into embedded sub-documents as well. In the
above example, Martina Navratilova didn't just compete as a single
player, so we might want to structure her record as follows:

<img src="https://mongodb-devhub-cms.s3.us-west-1.amazonaws.com/old_images/Polymorphic2-bpk0w3ig5y.gif" width="760" alt="Polymorphic Design Pattern with sub-documents" />

From an application development standpoint, when using the Polymorphic
Pattern we're going to look at specific fields in the document or
sub-document to be able to track differences. We'd know, for example,
that a tennis player athlete might be involved with different events,
while a different sports player may not be. This will, typically,
require different code paths in the application code based on the
information in a given document. Or, perhaps, different classes or
subclasses are written to handle the differences between tennis,
bowling, soccer, and rugby players.

## Sample Use Case

One example use case of the Polymorphic Pattern is [Single View
applications](https://www.mongodb.com/use-cases/single-view). Imagine
working for a company that, over the course of time, acquires other
companies with their technology and data patterns. For example, each
company has many databases, each modeling "insurances with their
customers" in a different way. Then you buy those companies and want to
integrate all of those systems into one. Merging these different systems
into a unified SQL schema is costly and time-consuming.

[MetLife](https://www.metlife.com/) was able to leverage MongoDB and the
Polymorphic Pattern to build their single view application in a few
months. Their Single View application aggregates data from multiple
sources into a central repository allowing customer service, insurance
agents, billing, and other departments to get a 360° picture of a
customer. This has allowed them to provide better customer service at a
reduced cost to the company. Further, using MongoDB's flexible data
model and the Polymorphic Pattern, the development team was able to
innovate quickly to bring their product online.

A Single View application is one use case of the Polymorphic Pattern. It
also works well for things like product catalogs where a bicycle has
different attributes than a fishing rod. Our athlete example could
easily be expanded into a more full-fledged content management system
and utilize the Polymorphic Pattern there.

</div>

<div class="summary">

## Conclusion

The Polymorphic Pattern is used when documents have more similarities
than they have differences. Typical use cases for this type of schema
design would be:

-   Single View applications
-   Content management
-   Mobile applications
-   A product catalog

The Polymorphic Pattern provides an easy-to-implement design that allows
for querying across a single collection and is a starting point for many
of the design patterns we'll be exploring in upcoming posts. The next
pattern we'll discuss is the
[Attribute Pattern](/developer/products/mongodb/attribute-pattern/).

</div>
