Copy atlas collection to another database

MongoDB Atlas
, ,
6

Hi @Benjamin_Flast thanks again.
Looks like we’ll take the DataLake route using a trigger with $merge or $out.

I have one last question around DataLake & Atlas triggers.

Is there a way to setup DataLake or Atlas triggers locally or in a Docker container?
We currently have a mongodb-memory-server that gets spun up for our integration tests. I’m hoping to do something similar for this scheduled task that will copy / update the analytics db. My thinking isn’t to test the triggers or mongo functionality; I assume you have that covered.
I mainly want to have a canary test to indicate if something may be broken and ensure our queries are correct so we can catch things early. For example if we have a schema change that may affect the aggregate query or something like that.

In case someone else stumbles across this thread searching for the same thing our approach as a rough/ pseudocode-ish overview on the same db using the mflix movies example collection is as follows:

  1. Link DataLake to Db
  2. Create Atlas Trigger to run every 24 hours
    Example trigger function below. 
exports = async function() {

  const movies = context.services
    .get("DataLake0")
    .db("Database0")
    .collection("Collection0");
    
    const pipeline = [
      {
            $match: {}
      }, {
    "$out": {
      "atlas": {
        "projectId": "111111111111111111111111",
        "clusterName": "mflix",
        "db": "analytics",
        "coll": "test"
      }
    }
}
   ];
  return movies.aggregate(pipeline);
};