We use mongodb for our production database. We are hoping to improve test reliability & isolation by having each test worker on a machine talk to its own db in mongodb.
So, we are trying to spin up 30 identical dbs on our test server. Each db should start out with empty collections, but the collections need to have indexes & validation rules defined on them.
We tried to do this in two ways:
A node script, which uses the mongodb nodejs driver to connect to each db and call the
createIndexand various other commands.
Preparing a single correctly configured db, dumping it via
mongodump, then replicating it to the other 29 dbs via
Both of these approaches fail to parallelize. While doing the setup on a single db takes ~2s, doing it across 30 dbs takes over 40s. In the case of option 1, we were seeing very inconsistent index creation times, ranging from 40s to over 3 minutes. Since the dbs are independent of each other, we expect this to take a similar amount of time as a single db, so 2-5s.
It appears that the mongo instance running on the test server is not able to run index creation across several dbs at the same time. Is this a known limitation? Are we mis-configuring mongo in some way that prevents it from running these operations in parallel?
Thanks for your input!
FYI, the dump/restore command we are using is:
# ... set up db test-1 (takes ~2s) mongodump --archive="test-1" --db=test-1 > test_output/mongodump-log 2>&1 echo "Setting up dbs via mongorestore" # this takes 40s. Why? seq 2 $NUM_SERVERS | xargs -L 1 -I % -P $NUM_SERVERS sh -c '\ mongorestore --archive=test-1 test-% --nsFrom="test-1.*" --nsTo="test-%.*" > test_output/mongorestore-test-% 2>&1; \ ' echo "Done setting up dbs"