Docs Menu

Docs HomeMongoDB Atlas Data Lake

View Atlas Data Lake Pipelines

You can view all of your Data Lake pipelines through the Atlas UI and Data Lake Pipelines API. You can also retrieve all of your completed Data Lake pipeline data ingestion jobs from the API.

1
2

The page displays all the Data Lake pipelines in the project. For each Data Lake pipeline, the service also displays the following information:

Column Name
Description
Pipeline Name
Name of your Data Lake pipeline. Each pipeline can produce multiple datasets. You can expand the name to view the datasets in the pipeline.
Data Source
Source for the data in the pipeline datasets. For data from a collection on the Atlas cluster, this column shows the cluster name, the database name, and the collection name separated by |.
Data Size
Size of data for each dataset.
Last Run Time
Date and time when the pipeline ran to ingest data for each dataset.
Status

Status of the pipeline. Value can be one of the following for a pipeline:

  • Active - indicates that the pipeline is active

  • Paused - indicates that data ingestion for the pipeline is paused

Frequency
Frequency at which cluster data is ingested and stored for querying.
Actions

Actions you can take for each pipeline. You can click one of the following:

  • || to pause data ingestion and to resume data ingestion. You can't pause on-demand ingestion of data.

  • to edit the data ingestion schedule for the pipeline.

  • to do the following:

    • Delete a pipeline. You can't undo this action. If you delete a pipeline, Atlas Data Lake deletes the datasets, including the data, and removes the datasets from the Federated Database Instances where they are referenced. If you delete a dataset inside a pipeline, Atlas Data Lake removes the dataset from the Federated Database Instance storage configuration where the dataset is referenced.

    • Trigger an on-demand pipeline run.

To retrieve all your Data Lake pipelines for a project through the API, send a GET request to the Data Lake pipelines endpoint. To learn more about the pipelines endpoint syntax and parameters for retrieving all of your Data Lake pipelines, see Return All Data Lake Pipelines from One Project.

To retrieve one of your Data Lake pipelines through the API, send a GET request to the Data Lake pipelines endpoint with the name of the Data Lake pipeline that you want to retrieve. To learn more about the pipelines endpoint syntax and parameters for retrieving one of your Data Lake pipelines, see Return One Data Lake Pipeline.

To retrieve all the completed Data Lake pipeline data ingestion jobs for a project through the API, send a GET request to the Data Lake runs endpoint. To learn more about the API syntax and options for the runs endpoint, see Return All Data Lake Pipeline Runs from One Project.

To retrieve the details of one of your completed Data Lake pipeline data ingestion jobs through the API, send a GET request to the Data Lake runs endpoint with the unique identifier of the completed Data Lake pipeline data ingestion job that you want to retrieve. To learn more about the API syntax and options for the runs endpoint, see Return One Data Lake Pipeline Run.

←  Manage Atlas Data Lake PipelineEdit an Atlas Data Lake Pipeline →
Share Feedback
© 2023 MongoDB, Inc.

About

  • Careers
  • Investor Relations
  • Legal Notices
  • Privacy Notices
  • Security Information
  • Trust Center
© 2023 MongoDB, Inc.