Q. mongodump with TTL index

Is there a possibility of data loss when running mongodump for Collections with TTL index?

I am using mongodb&mongodb tools 4.2.12 version.

Whats your concern? I suppose there won’t be a loss but you can have a verification check easily with an ttl index

Hi, @Kobe_W.
My concern is default.

And commands such as fsyncLock() are not possible because it is a commercial service that requires CUD commands of 1K or more to come in continuously even while running mongodump.
Therefore, it is impossible to confirm whether the backup was completely successful without data loss.
However, there were cases where some backups could determine that there was a data loss beyond the margin of error.

How about using a secondary node for backup purpose only? (e.g. hidden). We are using this approach in our current clusters. (lock the backup node → back up → unlock the node → verify ops log has no new entries, meaning no writes happen during the process)

That being said, my personal opinion is that, backup is mostly for “disaster recovery/mitigation”. The best is always to avoid a disaster. This is because backup normally happens at one time point in the past (unless you completely stop the writes to the cluster), so the changes between that time point and the current time point will be lost. As a result, someone would have to go through ops log and try to apply those changes manually. But this can be very challenging in a sharded cluster, especially when cross shard transaction happens.

1 Like

Hi @Kim_Hakseon, if you use mongodump with --oplog that will provide a consistent point-in-time backup of the cluster. This will provide a point-in-time backup of collections with TTL indexes. You can then use mongorestore with --oplogReplay to restore the point-in-time backup. To use mongodump with --oplog, you must create a full dump of a replica set member, you cannot dump just specific collections.

Documents could be immediately deleted on the destination if their TTL has expired since the dump was taken. But you can always turn off the TTL using server parameters, restore into a pre-existing collection which doesn’t have the TTL index, or use --noIndexRestore to stop TTL indexes from being built on the destination.

--oplog only works with replica sets. If you want a point-in-time backup of a sharded cluster you will need to use fsyncLock or another method like disk snapshots.

See these documentation pages for more info on:

mongodump --oplog: https://www.mongodb.com/docs/database-tools/mongodump/#std-option-mongodump.--oplog
mongorestore --oplogReplay: https://www.mongodb.com/docs/database-tools/mongorestore/#std-option-mongorestore.--oplogReplay
The ttlMonitorEnabled server parameter: https://www.mongodb.com/docs/manual/reference/parameters/#mongodb-parameter-param.ttlMonitorEnabled
Options for backing up and restoring sharded clusters: https://www.mongodb.com/docs/manual/administration/backup-sharded-clusters/

Also, I’d recommend using the most recent version of the database tools. That’s v100.9.4 at the time of writing. The version bundled with 4.2 is quite old and unsupported. v100.9.4 of the database tools will still work with v4.2 of the server.

And just as an FYI v4.2 of MongoDB has been EOL since April 2023. See here for EOL dates: MongoDB Software Lifecycle Schedules | MongoDB

Hi, @Tim_Fogarty :smiley:
I am doing dump & restore using the option you mentioned.
However, a problem like the text occurred, so I posted a question on the community.

I tested the 6.0 version and the corresponding utility tool, and it was the same.

I’ll have to think about the next best option.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.