Why mongodump or mongorestore so slow?

Working with Data Developer Tools
,
1

Hello, when I use mongodump to backup data or mongorestore to restore data, it always take too much time, is there any way to reduce the cost of time?

mongod version: v4.4.8
data size: 371GB
restore time: about 5.5 hours
OS version: Ubuntu 20.04.2

2

Hi @zhou_yi,

Can you share some more details:

  • How much RAM do you have available for mongod?

  • What sort of storage are you using?

  • What sort of deployment are you backing up (standalone, replica set, or sharded cluster)?

  • Is 371GB the uncompressed data size or the compressed storage size (size on disk)?

mongodump has to read and uncompress all data via the mongod process, so system resources will be limiting factors in performance. If you have CPU to spare, compressing the output with --gzip can reduce the amount of I/O for writing the dump to disk. You can also reduce I/O contention by dumping to a separate disk from the one hosting your dbPath and pausing writes to your deployment while taking the mongodump.

mongorestore has to recreate all data files and rebuild indexes, so available resources will again be a limiting factor. If your resources are not maxed out, you could try adjusting concurrency via --numParallelCollections=<int> (default is 4) and --numInsertionWorkersPerCollection=<int> (default is 1).

If you are looking for a faster backup and restore approach, I suggest using filesystem snapshots or copying the underlying data files per Supported Backup Methods. These approaches do not require reading all of the uncompressed data into memory or recreating the data files, so are much more suited to backing up production deployments (especially where total data is significantly greater than available RAM).

Regards,
Stennie

3

Hi @Stennie_X , thanks for your reply, the details are as follows:

  • Available RAM for mongod : total available memory is 20GB, but I see the document mentioned that WiredTiger Engine will use 50% of (RAM - 1 GB), so it may be 9.5GB?

  • Storage type: WiredTiger

  • Deployment type: sharded cluster, but the router, config server, shard are both on a single server

  • 371GB is uncompressed data size

I will learn and try the approach in your suggestion, thanks a lot!

1 Like
4

If more than one mongod is running on a single server, each mongod will have far less RAM than

Each mongo components running on the same single server will each fight for the same limited set of resources. Making a system slow. Having more than 1 mongod running on the same server should only be done when testing.

3 Likes