Regards.
This is my first post and after using mongodb in production for about 6 years now we started using watchers and we have following issue.
We have 3 members of 1 replica set and we are using node.js driver and a watcher implemented like this:
const offerTreeTask = me.mdb.collection('offer_tree');
const offerTreeChangeStream = offerTreeTask.watch([]);
offerTreeChangeStream.on('change', (change) =>
{
me.shouldLoadTree = true;
});
A simple straightforward implementation, with 20 clients creating this watcher. What happens after some time is that primary in replica runs out of memory and no matter how much we increase the memory on that machine it happens.
If we remove watchers then everything runs fine.
Database itself is 3GB in size, tried it with 3.6, 4.0 and 4.2 versions all have same issue.
We tried 3 different implementations, first one to return document on change (fullDocument) this resulted in running out of memory in a day on a 8GB machine, second one we changed implementation to just notify on change then it runs out in 2 days and third one we tried to close connection and reconnect every 5 minutes, this is the worst as then it runs out in half a day.