As I understand it you have:
- An “original” collection with documents, and new documents get inserted at some regular basis?
- Your goal is to have a separate collection, with documents that have a different structure, based off the original documents? Hence the aggrgation?
Are those “original” inserts happening at any time, or at the hourly mark you mentioned? Or is the hourly when you planned to do this sync?
The exact performance really depends on your application as a whole, etc. so it is a little hard to say.
However you could use these concepts to guide your decision:
If you are using Atlas you could use either Database Triggers or Scheduled Triggers.
The main difference being the Database Triggers will fire when a document is added or in some way modified … so performance here will mostly be around what your trigger code does, and if doing that per document is best. This has the advantages of keeping things up-to-date with the original collection automatically, but the downside of it might run very often, depending on how many modifications you have.
The Scheduled Triggers can let you batch together an operation, so for example you can run a single aggregation with a $merge at the end to mass insert any new data, and run that on an hourly schedule.
If you aren’t using Atlas you could still trigger something (like a cron job) to run the aggregation pipeline every hour as well.
Or, you can as part of your application code do this aggregation and insertion into the your aggregated collection.