We use mongodb for our production database. We are hoping to improve test reliability & isolation by having each test worker on a machine talk to its own db in mongodb.
So, we are trying to spin up 30 identical dbs on our test server. Each db should start out with empty collections, but the collections need to have indexes & validation rules defined on them.
We tried to do this in two ways:
A node script, which uses the mongodb nodejs driver to connect to each db and call the createIndex and various other commands.
Preparing a single correctly configured db, dumping it via mongodump, then replicating it to the other 29 dbs via mongorestore.
Both of these approaches fail to parallelize. While doing the setup on a single db takes ~2s, doing it across 30 dbs takes over 40s. In the case of option 1, we were seeing very inconsistent index creation times, ranging from 40s to over 3 minutes. Since the dbs are independent of each other, we expect this to take a similar amount of time as a single db, so 2-5s.
It appears that the mongo instance running on the test server is not able to run index creation across several dbs at the same time. Is this a known limitation? Are we mis-configuring mongo in some way that prevents it from running these operations in parallel?
Thanks for your input!
FYI, the dump/restore command we are using is:
# ... set up db test-1 (takes ~2s)
mongodump --archive="test-1" --db=test-1 > test_output/mongodump-log 2>&1
echo "Setting up dbs via mongorestore"
# this takes 40s. Why?
seq 2 $NUM_SERVERS | xargs -L 1 -I % -P $NUM_SERVERS sh -c '\
mongorestore --archive=test-1 test-% --nsFrom="test-1.*" --nsTo="test-%.*" > test_output/mongorestore-test-% 2>&1; \
echo "Done setting up dbs"
As a test, I ran this procedure with an empty db mongodump (no docs or indexes) and it succeeded nearly instantly, even with 30 mongorestores. So it does seem to be something related to index creation (even for an empty collection).
You do not seem to start mongorestore in background so your script is starting 39 mongorestore processes, one after the other, with no parallelism at the source. So even if mongod could do it in parallel your script is not.
So, we are trying to spin up 30 identical dbs on our test server.
How many mongod processes are we talking about? Are all 30 dbs live in a single mongod instance? If you have 30 mongod instances, are they on a single machine, or on separate machines?
If all 30 databases live in a single mongod process or if you have 30 mongod processes in a single machine, I don’t think you can expect it to run with the same timings as preparing a single database. This is because to create an index, it would need to do a collection scan which involves reading the whole collection into the cache. Multiply this by 30, and you’re hammering the cache and the disk with IO requests. If your disk cannot handle the requests, mongod will be forced to sit idle while waiting for the disk to complete its work. You can check if disk is the bottleneck by checking the iostat numbers, and see if the disk is fully utilized during this process.
Actually you can do a small experiment by trying this process with less parallelization numbers. Say start with 2 processes simultaneously, and observe the reported timings. Then gradually increase the number of parallelization until you don’t see the benefit of adding more processes. I would be interested in seeing at what point the hardware start to get overworked.
If you need further help, please post more details:
How much data is in the database, how many collections, and how many indexes
What is your MongoDB version
What is the spec of your OS, hardware, and how you run the mongod process
Please provide as much examples as possible so your experiment can be replicated and your timings verified by another person
Hey Kevin, thanks for the reply. We are looking into your suggestions and running some experiments.
A few clarifications:
We’re running 30 dbs on a single mongod process, which is running on the same machine.
The db setup we’re trying to achieve is to have indexes over empty collections. So I wouldn’t expect a collection scan to be an expensive operation. We looked at iostat and we don’t seem to be running against disk bottlenecks.
We have about 30 (empty) collections, with about 60 indexes total.
We’re using mongo 3.6.3 (same version for the shell and mongod)
We’re running this on an EC2 instance - m5.8xlarge so vCPU=32, Memory=128GiB. We’re using ubuntu, so mongod is started with systemd. We already looked into ulimit documentation and systemd is configured to start the service with the recommended settings.
We ran the parallelization experiment, and we’re getting about the same amount of time to start 30 dbs with parallelization 1 as with parallelization 5 or 30. (Which is leading us to suspect that there’s some locking in the mongod process that’s preventing it from creating these dbs in parallel).
We tried this approach (creating indexes via the js driver). It seemed to have the same bottleneck - and I don’t doubt that you would find the times increasing if you created 30 collections with 60 indexes per db.
This approach also experiences the same variability in performance, with some runs taking significantly more than 30s.