Wall field in oplog collection

Working with Data Developer Tools
,
#1

If I query the oplog collection, i can see a field named “wall”:

{
 	"ts" : Timestamp(1628594571, 1),
 	"h" : 0,
 	"v" : 2,
 	"op" : "n",
 	"ns" : "",
 	"wall" : ISODate("2021-08-10T14:22:51.547+03:00"),
 	"o" : {
 		"msg" : "initiating set"
 	}
}

When I turn the timestamp of the wall field into Epoch, or turn the “ts” number into a timestamp, it looks like they give the same time.

Could not find any explanation on this field though.

I was wondering if I can use the wall field in order to extract the required ts for a point-in-time recovery.

#2

To give an example, this is the query I was thinking about:

db.oplog.rs.find({“wall”:{ “$gt” : new ISODate(“2021-08-24T00:00:00.000Z”)}},{“ts”: 1,“wall”:1}).sort({“wall”:1}).limit(1);`

The resut would be:

{
	"ts" : Timestamp(1629763200, 1),
	"wall" : ISODate("2021-08-24T03:00:00.654+03:00")
}
#3

Hi @Tamar_Nirenberg

The ts field in the oplog is seconds since epoch and then order of inserts in that second.

Wall time is added in https://jira.mongodb.org/browse/SERVER-27769

Change oplog OpTime format to raw 64bit values

So it is likely a part of that transition as the seconds since epoch is a 32bit integer that will max out in 2038

Sure I don’t see why not. I still use the ts field at the moment.

1 Like
#5

Hi @Tamar_Nirenberg

I think @chris has given the best answer regarding this field.

However I would like to add that as you have discovered, the oplog collection is designed for internal MongoDB use, so things can change unexpectedly between versions. I would not depend on the oplog for operational purposes.

Depending on your needs, Change Streams is the supported methods to access real-time data changes that’s more accessible and consistent. It even works across sharded clusters (oplog is limited to a single replica set).

Regarding backup methods, the supported methods are outlined in MongoDB Backup Methods.

Best regards
Kevin

1 Like
#7

Hi @kevinadi

Thanks fr the reply.
“MongoDB Backup Methods” does say the following under the section:

Back Up with `mongodump

For replica sets, mongodump provides the --oplog option to include in its output oplog entries that occur during the mongodump operation. This allows the corresponding mongorestore operation to replay the captured oplog. To restore a backup created with --oplog , use mongorestore with the --oplogReplay option.

I realize other methods are more secure, but unfortunatlly I do not have an approval for using other backup methods which are licensed.

#8

Hi @Tamar_Nirenberg

To be clear, the --oplog option was designed into mongodump so that once the backup starts, changes to the databases are also recorded at the same time via the oplog. This is to allow mongorestore to restore the underlying data, while also making the restored instance up to date once the restore is finished.

Although with some gymnastics it may be able to provide a custom point-in-time restore by specifying --oplogLimit, it was not its primary design goal.

Notably, using the --oplog option does not allow you to backup/restore specific databases or collections. --oplog requires you to perform mongodump for the instance as a whole.

Best regards
Kevin

#9

Hi @kevinadi,

Just to make sure I understand:
If I perfomr mongodump at a certain time, then mongorestore will restore data inserted to the DB while the mongodump was still running?

That is - say I started the mongodump at 22:00, and it finished running at 23:00 - If I use mongoprestore to restore the data to another mongo instance, it will include all the changes done on the original DB between 22:00 and 23:00?

Thanks,
Tamar

#10

Hi @Tamar_Nirenberg

Yes, as mentioned in the mongodump --oplog manual:

Creates a file named oplog.bson as part of the mongodump output. The oplog.bson file, located in the top level of the output directory, contains oplog entries that occur during the mongodump operation. This file provides an effective point-in-time snapshot of the state of a mongod instance. To restore to a specific point-in-time backup, use the output created with this option in conjunction with mongorestore --oplogReplay .

Where I believe this paragraph directly answers your question:

Without --oplog , if there are write operations during the dump operation, the dump will not reflect a single moment in time. Changes made to the database during the update process can affect the output of the backup.

Best regards
Kevin

3 Likes
#11

Thank you @kevinadi for the clarification!