Is this right way to clean data in MongoDB?

Working with Data
1

Hello! We want to clear our Mongo of old records (older than 90 days), we have a replica set with 3 nodes and we have 3.8 terabytes of data (in /var/lib/mongodb) on each node

We want to clear old records using this way

db.Clicks.deleteMany({ “callData.path”: “/info-dubai”, “callData.date”: { $lt: new Date(new Date().getTime() - (86400000 * 90)) } } );

db.Clicks.deleteMany({ “callData.path”: "/info-qatar, “callData.date”: { $lt: new Date(new Date().getTime() - (86400000 * 90)) } }) ;

And after completing these two requests, disk space is not freed up, and i have to run the command for compaction

db.runCommand( { compact: ‘Clicks’ } );

After compaction, the space is freed up, but the cleaning itself takes a very long time and loads I/O quite heavily, and compaction itself also affects performance, please tell me, maybe there is a way to do this more productively? faster and with less load

2

Do it in batch maybe.

Also, do you have proper index on them? how many rows are in scope?

Compaction will definitely take lots of resources.

3

If documents are going to be added again there is little point to compacting, the blocks freed in the collection from the deletes will be used by future inserts and updates.

  • Don’t do it, as explained above
  • Look into using a TTL index to constantly maintain the 90 day window.

Doing a compact on a replicaset should not impact performance as this should be performed in a rolling maintenance fashion. Execute on each secondary waiting for each to complete before starting the next, step down the primary, then execute on the what was previously the primary.

https://www.mongodb.com/docs/manual/core/index-ttl/
https://www.mongodb.com/docs/manual/reference/command/compact/#replica-sets

1 Like