Rollback with {w: majority, j: true}

I am developing an application that uses a sharded mongo cluster… All writes use a {w: majority, j: true}.

At some point, our cluster auto-scaled to increase disk space and several of our nodes experienced rollback events.

How can a rollback take place when all writes use {w: majority, j: true}?

My current best guess of what could have happened is:

  1. Application sends a write to the primary with {w: majority, j: true}
  2. The primary writes to its oplog and journal
  3. The primary crashes before the write has been replicated to a majority of voting members.
    • The application would not receive acknowledgement of the write in this case.
  4. An election takes place and a secondary becomes primary.
  5. The crashed primary re-joins as a secondary. Since the other two members do not have the write, a rollback occurs.