Data Source Pipelines
On this page
- Use Cases
- Add a Pipeline to a Data Source
- Open the Data Sources view.
- For the data source you wish to pre-process, click Add Pipeline in the Applied Aggregations column.
- In the Edit Aggregation Pipeline modal, create your aggregation pipeline.
- Click Save.
- Example Pipelines
- Join Data from Two Collections
- Hide Sensitive Data
- Convert the Type of a Data Field
- Modify an Existing Data Source Pipeline
- Open the Data Sources view.
- Open the pipeline modification dialog.
- In the Edit Aggregation Pipeline modal, make the desired changes to your pipeline.
- Click Save.
- Delete an Existing Data Source Pipeline
- Open the Data Sources view.
- For the pipeline you want to modify, click the Gear icon.
- Click Delete Pipeline.
- Click Delete to confirm.
If you are the Owner of a data source, you can
apply an aggregation pipeline
to a data source to pre-process data before it reaches the
chart builder.
Use Cases
Pre-processing a data source with a pipeline is useful when your collection data is not yet optimized for data visualization. While you can still pre-process your data by creating a database view or using the Charts query bar, adding a pipeline to a data source ensures that any charts using the data source receive the same data transformations.
Pre-processing your data can help accomplish the following tasks:
- Join data from multiple collections into a single data source using
$lookup. See example. - Hide fields which may contain sensitive data from Charts users
using
$project. See example. - Convert data types, such as a string field to a date. See example.
Add a Pipeline to a Data Source
Open the Data Sources view.
Click Data Sources in the MongoDB Charts sidebar.
For the data source you wish to pre-process, click Add Pipeline in the Applied Aggregations column.
In the Edit Aggregation Pipeline modal, create your aggregation pipeline.
Your pipeline must be an array in square brackets. Specify each stage of your pipeline as an object within the array.
MongoDB Charts validates your pipeline before you save. Charts reports any errors in your pipeline below the editor.
For help creating your pipeline, you can utilize the Aggregation Pipeline Builder in the Atlas Data Explorer.
You can create your pipeline using the Atlas pipeline builder and paste it into the MongoDB Charts pipeline editor.
Click Save.
Example Pipelines
Join Data from Two Collections
The following example adds a pipeline to a data source pointing to
the sample_mflix.movies collection from the
Sample Mflix Dataset provided by
Atlas. The pipeline performs a $lookup to join data
from the comments collection based on movie_id. The result is
a new field in this data source called comments containing data
from a separate collection.
[ { $lookup: { from: 'comments', localField: '_id', foreignField: 'movie_id', as: 'comments' } } ]
Hide Sensitive Data
The following example adds a pipeline to a data source pointing to the
sample_supplies.sales collection from the
Sample Supply Store Dataset
provided by Atlas. The pipeline hides the customer.email field,
making that field unavailable to charts using this data source:
[ { $project: { "customer.email": 0 } } ]
To learn more about the $project aggregation stage,
see $project.
Convert the Type of a Data Field
The following example adds a pipeline to a data source containing a
string field that we want to convert to a date. The pipeline adds a new
field to the data source with the string field saleDate converted to
a date field called saleDateAsDate:
[ { $addFields: { saleDateAsDate: { $toDate: '$saleDate' } } } ]
Modify an Existing Data Source Pipeline
You can only modify pipelines on data sources of which you are
an Owner.
Charts displays a warning if the data source's pipeline you attempt to modify is in use by at least one chart. Modifying a pipeline which is currently in use may result in charts using the pipeline rendering incorrectly or breaking entirely.
Open the Data Sources view.
Click Data Sources in the MongoDB Charts sidebar.
Open the pipeline modification dialog.
- For the pipeline you want to modify, click the Gear icon.
- Click Edit Pipeline.
In the Edit Aggregation Pipeline modal, make the desired changes to your pipeline.
Your pipeline must be an array in square brackets. Specify each stage of your pipeline as an object within the array.
MongoDB Charts validates your pipeline before you save. Charts reports any errors in your pipeline below the editor.
For help creating your pipeline, you can utilize the Aggregation Pipeline Builder in MongoDB Compass.
You can create your pipeline using Compass's pipeline builder and paste it into the MongoDB Charts pipeline editor.
Click Save.
Delete an Existing Data Source Pipeline
You can only delete pipelines on data sources of which you are
an Owner.
Charts displays a warning if the data source's pipeline you attempt to delete is in use by at least one chart. Deleting a pipeline which is currently in use may result in charts using the pipeline rendering incorrectly or breaking entirely.
Open the Data Sources view.
Click Data Sources in the MongoDB Charts sidebar.