We have a Mongo DB Atlas service, and in AWS we have an s3 bucket with many .csv files. We need to load this csv data into our Mongo DB database collection every 5 minutes. We have been reviewing documentation, but we have not been able to achieve this. What can we do? Can someone please provide us with documentation or videos on how to do this?
Hello!
Atlas Data Federation enables you to query CSV files stored in S3 buckets using MQL. It also provides the $out command to write the results of an aggregation pipeline to a specified collection. Using Data Federation, you can construct a pipeline that reads your CSV files, performs any operation needed, and writes the results into the active cluster of your choice.
Details on Using Atlas Data Federation can be found here: https://www.mongodb.com/docs/atlas/data-federation/
Thanks!
Irwin