Welcome to the MongoDB Community @Db_learner !
- if we take backup using mongodump before migration(with mmapv1 on all nodes), can that be used for restoring data when all nodes are on wiredtiger ?
mongodump taken from an MMAP deployment can be restored into one using WiredTiger.
A general caveat on restoring old backups is that there have been improvements in validation of collection and index options in successive versions of MongoDB, so if your backup is taken from an older server release (eg taken from MongoDB 3.2, restoring into 4.0) you may have some minor issues to resolve.
- Can both storage engines coexist in same cluster with wiredtiger as primary and one of the secondary nodes with mmapv1 for some time, so that in case of a rollback necessity, we can just let the sync happen from secondary with mmapv1 to rest of nodes getting switched back to mmapv1.
Yes, the recommended approach to change your storage engine is via a rolling upgrade with mixed storage engines: Change Replica Set to WiredTigerChange Replica Set to WiredTiger (MongoDB 3.6). You can use the same approach to change from WiredTiger to MMAP if needed.
However, since MMAPv1 was deprecated and ultimately removed in the 4.2 release I recommend investigating any WiredTiger issues before rolling back. MongoDB 3.6 was first released in Nov 2017 and reached End of Life in April, 2021. There have been four major release series since 3.6 so far, with many improvements.
I also would not recommend running with mixed storage engines for an extended period of time as differences between storage engine behaviour may have operational impact. For example, MMAP doesn’t support
majority read concern which is required for the new Change Streams feature in MongoDB 3.6.