Hi everyone,
I am setting up Data Federations and want to move data from cluster to S3 following guidance from How to Automate Continuous Data Copying from MongoDB to S3. I have setup my cluster and s3 as Data Sources in Data Federations. Next, I have created Linked Data Sources to service name “kaison-data-federation” for Data Federations.
Next, in my triggers, I wrote below functions:
exports = function () {
const datalake = context.services.get(“kaison-data-federation”);
const db = datalake.db(“targeting”)
const events = db.collection(“subscriptions”);
const pipeline = [
{
$match: {
$and: [
{ status: ‘active’ }
]
}
},
{
“$out”: {
“s3”: {
“bucket”: “test-mongo-atlas-data-lake”,
“region”: “ap-southeast-1”,
“filename”: “events”,
“format”: {
“name”: “parquet”,
“maxFileSize”: “10GB”,
“maxRowGroupSize”: “100MB”
}
}
}
}
];
return events.aggregate(pipeline);
};
However, I am hitting error of “(Unauthorized) not authorized” with no exact error mentioning. I have refer to other posts like Unauth Error on Moving Data from DataLake to S3. But still not working. I have stucked here for 3 days. Can anyone provide some insight to me on this? Thank you very much!
