Hi @Trdat_Mkrtchyan,
There are a number of possible solutions to that, one possibility, for example:
- Have a database trigger on the collection you want to observe: the connected function does nothing but keeping track of the
$clusterTime(i.e. the time the change was applied to the DB), and save it somewhere if higher than the previous one. - Have a scheduled trigger, running every 5-10 minutes, that observes the latest registered time the collection was changed: if more than an hour has passed, than the collection is ready for cleanup, and a relatively small chunk of data can be processed
- Let the schedule cycle through all the chunks, until it’s done, and wait for the next update
- You can also add more complex logic, i.e. by saving different
$clusterTimefor each 3rd party, and process at any given time only the updates of the one that had finished first.
Does the above make sense?