we are using Mongo as our event store in our event sourced project.our database has only one replica just for supporting transactions and we are not gonna use multi replica, shards or any other advanced features of Mongo.
in order for projections of events into a report table idea to work, we need to be able to permanently resume “Change Streams” from any point in the history.and we just need the history of insertions, no more
to do that, we store resume token and also operation time of each insertion along with it’s id in a separate collection, and when we want to resume from a certain insertion in the history, we query and find the exact resume token and we resume it from there
this causes a bunch of problems:
1-storing resume tokens and operation time of each insertion via a change stream is fragile and also takes storage, but works!!!
2-we recently hit the oplog size and found out that resumability is possible with oplog so in order to be able to resume from any point in the history, we have to store and keep the whole oplog throughout the life of application, but the size of oplog hits it’s max capacity every 2 hours!
so it is not possible to save and keep oplog!
what should we do?
it seems there is no way other than implementing change stream from history ourselves on the Mongo.