Hello, I am currently working on a Python tool to move cold data from our production cluster to an S3 bucket using data federation. Like this tutorial MongoDB Atlas Data Federation Tutorial: Federated Queries and $out to AWS S3 I want to delete the cold data from our cluster, after the aggregation pipeline finished.
However, it seems that I do not get any indication from PyMongo if the pipeline finished successfully or if an error occured. As per Atlas documentation on the $out stage the error output is only written to the S3 bucket. So the delete could potentially drop data which has not been transfered, which is a huge no-go for our project.
I also could not find a way to get the <correlation-id> described in the documentation, so currently it seems that the only option would be to search the S3 bucket for any folder that matches the error folder path described in the documentation.
Is it possible to directly get a success/fail message from the aggregation pipeline?