Analyze oplog.rs collection to calculate space used by each name space

We have multiple databases in the replica and want to move one of the databases to another replica set. We want to be able to calculate how much space is occupied by this database in the OPLOG.RS collection in order to correctly size it in new replica set.

Welcome to the MongoDB community @Kamal_Parshotam !

The replication oplog only includes a rolling subset of data changes. Once the oplog collection reaches its maximum configured size, older oplog entries will be removed to keep storage usage within expectations.

If you want to find the size of a collection on disk, use the db.collection.stats().storageSize value as a guide.

Regards,
Stennie

1 Like

Thanks Stennie. For a more detailed breakdown of the documents in the collection, I managed to write this query

db.oplog.rs.aggregate([
{$group : {_id:{collection:"$ns", operation:"$op"}, “combined_object_size”: { $sum: { $bsonSize: “$$ROOT” } } , DocCount:{$sum:1} }}
])

This allowed me to do some slicing and dicing of the result set.

Hi @Kamal_Parshotam,

Since oplog entries only include document insertions and changes (and a rolling history), this may not be a useful indication of actual document size. A $group query on the oplog will also be a full collection scan and may have some performance impact on a production system.

You could use db.collection.stats().avgObjSize to quickly get the average document size for a collection (similar to your query which is getting the BSON size of oplog entries), however the storage size would likely be much less with compression.

Regards,
Stennie

1 Like

Hi Stennie,

The issue we were facing was that the OPLOG WINDOW had gone up from 10 hours to 5 days. I needed to understand which namespace + operation was occupying the most space in the OPLOG.RS collection, hence the group by. We actually found that a NOOP operation in a particular database / collection was occupying 90% of the oplog window. This was quite revealing.

Anyway , thanks once again.

1 Like

Copieded! Thanks @Stennie_X

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.