Recently I have faced data loss in MongoDB, and that is so rare:
– checked for rollback, nothing there.
found that, only node in replicaset is alive, but unable to connect, on restart that single replica node, it starts replaying oplog, not sure if it is because member was not identified as primary or secondary.
My questions here:
does oplog replay truncate data from primary?
insights_bigd
I STORAGE [initandlisten] Sampling from the oplog between Jun 3 23:05:13:2256 and Jun 5 16:25:37:1 to determine where to place markers for truncation
No. Oplog replay does not remove any data from the primary.
The oplog is a special capped collection with a configured maximum size. When the maximum size of a capped collection is reached, some of the oldest documents are removed to maintain the target oplog size.
The oplog implementation in the WiredTiger storage engine has some optimisations specifically for replication use cases. A housekeeping process adds logical markers (aka oplog stones) for more efficient removal of batches of the oldest documents.
Ok, when does oplog replay comes to picture. Like we have primary node, but I can see oplog replay operations in primary log as well.
Also what if we drop “local” database from primary only node. does we get data loss, as the data is already written in disk, but I can see data loss after dropping “local” database
Can you clarify what you mean by “oplog replay”? Are you referring to a secondary pulling updates from another member of your replica set? Can you provide an example of the operations you are looking at?
Dropping the local database removes the collections and configuration it contains (startup_log, oplog.rs, system.replset, …). but does not affect data in other databases.
However, replica set members rely on the oplog to determine the provenance of data. If you drop the local database (or the local.oplog.rs collection) for a replica set member, it cannot rejoin the same replica set without resyncing all data. If a replica set member has an oplog but no longer has an entry in common with the oplog of another member of the replica set, a resync will also be required.
If there are data changes you want to preserve on a replica set member that you are about to resync, you should take a backup of the relevant data before starting the resync process. The resync process replaces all data and indexes with a fresh copy (and known state) from another replica set member.