Docs Menu

Docs HomeDevelop ApplicationsMongoDB Manual

Rollbacks During Replica Set Failover

On this page

  • Collect Rollback Data
  • Avoid Replica Set Rollbacks
  • Rollback Considerations

A rollback reverts write operations on a former primary when the member rejoins its replica set after a failover. A rollback is necessary only if the primary had accepted write operations that the secondaries had not successfully replicated before the primary stepped down. When the primary rejoins the set as a secondary, it reverts, or "rolls back," its write operations to maintain database consistency with the other members.

MongoDB attempts to avoid rollbacks, which should be rare. When a rollback does occur, it is often the result of a network partition. Secondaries that can not keep up with the throughput of operations on the former primary, increase the size and impact of the rollback.

A rollback does not occur if the write operations replicate to another member of the replica set before the primary steps down and if that member remains available and accessible to a majority of the replica set.

Starting in version 4.0, MongoDB adds the parameter createRollbackDataFiles to control whether or not rollback files are created during rollbacks.

By default, when a rollback occurs, MongoDB writes the rollback data to BSON files.

Note

Rollback Directory Change

Starting in Mongo 4.4, the rollback directory for a collection is named after the collection's UUID rather than the collection namespace.

If the operation to roll back is a collection drop or a document deletion, the rollback of the collection drop or document deletion is not written to the rollback data directory.

To read the contents of the rollback files, use bsondump. Based on the content and the knowledge of their applications, administrators can decide the next course of action to take.

For replica sets, the write concern { w: 1 } only provides acknowledgement of write operations on the primary. Data may be rolled back if the primary steps down before the write operations have replicated to any of the secondaries. This includes data written in multi-document transactions that commit using { w: 1 } write concern.

To prevent rollbacks of data that have been acknowledged to the client, run all voting members with journaling enabled and use { w: "majority" } write concern to guarantee that the write operations propagate to a majority of the replica set nodes before returning with acknowledgement to the issuing client.

Starting in MongoDB 5.0, { w: "majority" } is the default write concern for most MongoDB deployments. See Implicit Default Write Concern.

With writeConcernMajorityJournalDefault set to false, MongoDB does not wait for w: "majority" writes to be written to the on-disk journal before acknowledging the writes. As such, "majority" write operations could possibly roll back in the event of a transient loss (e.g. crash and restart) of a majority of nodes in a given replica set.

  • Regardless of a write's write concern, other clients using "local" or "available" read concern can see the result of a write operation before the write operation is acknowledged to the issuing client.

  • Clients using "local" or "available" read concern can read data which may be subsequently rolled back during replica set failovers.

For operations in a multi-document transaction, when a transaction commits, all data changes made in the transaction are saved and visible outside the transaction. That is, a transaction will not commit some of its changes while rolling back others.

Until a transaction commits, the data changes made in the transaction are not visible outside the transaction.

However, when a transaction writes to multiple shards, not all outside read operations need to wait for the result of the committed transaction to be visible across the shards. For example, if a transaction is committed and write 1 is visible on shard A but write 2 is not yet visible on shard B, an outside read at read concern "local" can read the results of write 1 without seeing write 2.

Starting in version 4.2, MongoDB kills all in-progress user operations when a member enters the ROLLBACK state.

For more information on the index build process, see Index Builds on Populated Collections.

Disabling "majority" read concern prevents collMod commands which modify an index from rolling back. If such an operation needs to be rolled back, you must resync the affected nodes with the primary node.

Changed in version 4.0.

Starting in version 4.0, MongoDB has no limit on the amount of data that can be rolled back.

In previous versions, a mongod instance will not roll back more than 300 megabytes of data and requires manual intervention if more than 300 megabytes of data need to be rolled back.

Starting in version 4.0, the rollback time limit defaults to 24 hours and is configurable using the parameter rollbackTimeLimitSecs:

  • In MongoDB 4.2+ and 4.0.13+, the rollback time limit is calculated between the first operation after the common point and the last point in the oplog for the member to roll back.

  • In MongoDB 4.0.0-4.0.12, the rollback time limit is calculated between the common point and the last point in the oplog for the member to roll back.

In MongoDB 3.6 and earlier, the rollback time limit is not configurable. For these versions, rollback is limited by the amount of data, with a maximum of 300 megabytes.

Tip

←  Replica Set ElectionsReplica Set Read and Write Semantics →