Is this right way to clean data in MongoDB?

Hello! We want to clear our Mongo of old records (older than 90 days), we have a replica set with 3 nodes and we have 3.8 terabytes of data (in /var/lib/mongodb) on each node

We want to clear old records using this way

db.Clicks.deleteMany({ “callData.path”: “/info-dubai”, “callData.date”: { $lt: new Date(new Date().getTime() - (86400000 * 90)) } } );

db.Clicks.deleteMany({ “callData.path”: "/info-qatar, “callData.date”: { $lt: new Date(new Date().getTime() - (86400000 * 90)) } }) ;

And after completing these two requests, disk space is not freed up, and i have to run the command for compaction

db.runCommand( { compact: ‘Clicks’ } );

After compaction, the space is freed up, but the cleaning itself takes a very long time and loads I/O quite heavily, and compaction itself also affects performance, please tell me, maybe there is a way to do this more productively? faster and with less load

Do it in batch maybe.

Also, do you have proper index on them? how many rows are in scope?

Compaction will definitely take lots of resources.

If documents are going to be added again there is little point to compacting, the blocks freed in the collection from the deletes will be used by future inserts and updates.

  • Don’t do it, as explained above
  • Look into using a TTL index to constantly maintain the 90 day window.

Doing a compact on a replicaset should not impact performance as this should be performed in a rolling maintenance fashion. Execute on each secondary waiting for each to complete before starting the next, step down the primary, then execute on the what was previously the primary.

https://www.mongodb.com/docs/manual/core/index-ttl/
https://www.mongodb.com/docs/manual/reference/command/compact/#replica-sets

1 Like