I’m an application developer and don’t have much experience managing MongoDB databases/clusters/etc, mostly just CRUD operations. That being said, I’ve been tasked with updating our MongoDB Atlas dev database from production. They live on separate Atlas projects.
We have many databases in each respective project, however I only need to migrate data from one of the production databases into its respective dev database. I need to migrate any indexes attached to each database collection as well. And, I only need to pull over a subset of the production data, not the entire set, for the database. Both projects are on M10 instances.
Given that I have little experience with this, I’m struggling to find documentation, blog posts, stack overflow questions, etc, probably because I don’t know what to search for. I’m hoping for either detailed guidance, or links to articles with detailed guidance. If I left out any pertinent info please let me know
I’d also like to note that while it’s ideal to be able to migrate a subset of data from production, if it’s easier to pull the entire set, that’s acceptable as well.
Hi Brandon, you can use mongodump/restore for this. First you would take a dump of the production data that you want, then you can restore it to your dev cluster. A dump will backup anyindexes that are in the collections and mongorestore will restore them (unless you tell it not too).
Another great feature of mongodump is that you can pass the --query flag so you can do a json query to find specific data that you want to take a dump of.
A typical mongodump string looks like
mongodump --host <host> --port <port> --db <db> -u <username> --authtenticationDatabase <auth db> --out <where you want the file saved>
This will backup an entire db, you can look at the other options on how to refine your data you backup.
Thanks @tapiocaPENGUIN for the answer. That’s exactly the path I’m headed down.
I’m copying over multiple collections from the database. It’s not clear in the documentation how I would use --query with multiple collections being included in the dump. How would I go about doing that?
The dump seems to work fine, but piping the results to mongorestore results in the following with no error messages and no changes to the dev database.
0 document(s) restored successfully. 0 document(s) failed to restore.
If the data in each collection is different enough that the same query can’t be used for both collections, I would try doing it one collection at a time (you can specify the namespace (db.collection). Although it’s less efficient it will allow you to get the specific data you need for each collection.
As for not having any documents to restore, I would try it without piping to mongorestore and see if any documents are successfully backed up in the dump.
Got error from options parsing: (AtlasError) getting non-bucket system collections is unsupported
Failed: (AtlasError) getting non-bucket system collections is unsupported
I can’t find anything through google or on mongo’s site explaining what a non-bucket collection is.
Edit time has past on the previous post so just adding more info…
The dump appears to work. I’m seeing gzipped bson and metadata files for the expected collections in the /dump/{prodDatabaseName} directory.
Still reading through it, but looking at the mongorestore and mongodump documentation I see this blurb…
When using mongorestore to load data files created by mongodump, be sure that you are restoring to the same major version of the MongoDB Server that the files were created from. For example, if your dump was created from a MongoDB Server running version 4.4.x , be sure that the MongoDB Server you are restoring to is also running version 4.4.x .
Prod is on v4.2.18 and dev is on v4.4.12. While they are on the same major version, could this be the issue with the minor versions being different?
@tapiocaPENGUIN I really appreciate your continued help on this. Thank you so much!
From the looks of the section you posted it looks like by same major version they mean 4.4.x or 4.2.x. This could be an issue.
I’m not sure about this one either…I wonder if it has to do with a security measure since you are doing a dump from Atlas. Are you only backing up dbs and collections that you created?
I’m able to connect to Atlas for the dump and restore just seem to be getting an error when mongorestore parses the options. Here’s the entire script and the output.
#!/usr/bin/env bash
ORG_NAME=$1
DEV_DATABASE="migration-test-dev"
PROD_DATABASE="$ORG_NAME-prod"
# Backup collection from production database
mongodump --uri="mongodb+srv://brandon:33StU%26%25AV8N%24@shared-prod.s0nu7.mongodb.net/$PROD_DATABASE" \
--excludeCollection=$ORG_NAME-org \
--excludeCollection=$ORG_NAME-users \
--gzip
# Restore development database from production backup
mongorestore --uri="mongodb+srv://brandon:pH%5EQ0CRob9%24x@shared-dev.lymbn.mongodb.net/$DEV_DATABASE" \
--nsFrom="$PROD_DATABASE.*" \
--nsTo="$DEV_DATABASE.*" \
--drop \
--preserveUUID \
--gzip \
--verbose \
dump/$PROD_DATABASE
mongodump creates the following directory and files…
mongorestore outputs this to the terminal. I’ve added hyphens to the start of each line to make it more readable.
– Using write concern: &{majority false 0}
– Will listen for SIGTERM, SIGINT, and SIGKILL
– Connected to node type: replset
– The --db and --collection flags are deprecated for this use-case; please use --nsInclude instead, i.e. with --nsInclude=${DATABASE}.${COLLECTION}
– Got error from options parsing: (AtlasError) getting non-bucket system collections is unsupported
– Failed: (AtlasError) getting non-bucket system collections is unsupported
– 0 document(s) restored successfully. 0 document(s) failed to restore.
I was able to solve the issue of not being able to get mongorestore to work. Here’s a link to the question/answer in case someone else ends up with the same problem.
