Hello,
I have been reviewing this tutorial for “How to Automate Continuous Data Copying from MongoDB to S3” and would like to better understand how it works from a resiliency perspective.
In the example, the realm function uses an aggregation pipeline to find all documents created in the last 60 seconds, and sends them all to S3. The function is then called by a trigger that runs every 60 seconds.
What would happen in the scenario that the function failed to successfully execute? (ie S3 is unavailable, etc). Am I correct in my understanding that all documents created during the outage would never be sent to S3 as there is no queueing mechanism or use of the op log? For my use cause, failing to copy even a single document to S3 is unnacceptable, so I’m curious if it’s possible for this solution to work for me.
Thanks!
Christian
it’s more like a proof of concept that this is possible. It’s also creating a new parquet file every minute (which contains the last minute of data in MongoDB), which may or may not be what you want. If this has been running for a while, I imagine you’ll have a lot of files in that bucket.