Missing Indexes while Mongo restoration

Working with Data
,
1

I am doing a Backup of Data from Production to Lower Environments using mongodump and mongorestore, In the process of restoration, The Indexes are not getting restored and in the log file it shows the below output. Also the creation of remaining Indexes are also getting hold, Is there a way to make

2

Hi @Dinesh_Reddy_Sanikommu,
I ran a test and I get indexes restored correctly as well. Can you better explain what problem you are experiencing? What version of MongoDB are you working on?

Best Regards

3

Hi @Fabio_Ramohitaj , Thanks for the reply,

So when I clone the data (mongo dump) from Prod and restore in my Lower Environments (mongo restore), I believe there are few Indexes missing, Is this an expected behavior.

If not, please let me know what are validation steps should if perform?

Regards,
Dinesh

4

Also My Atlas Mongo DB version is 5.

5

Hi @Dinesh_Reddy_Sanikommu,

I simulated the following situation:

  1. Created the collection with some document:
db.collection.insertMany([{name: "Fabio"},{name: "Marco"},{name: "Luca"}])
{
 acknowledged: true,
 insertedIds: {
   '0': ObjectId('66acb407fe3ec6f3c4079058'),
   '1': ObjectId('66acb407fe3ec6f3c4079059'),
   '2': ObjectId('66acb407fe3ec6f3c407905a')
 }
}
  1. Create an index on the field name:
db.collection.createIndex({name:1})
  1. Dump the database with the mongodump utility:
mongodump -d database
  1. the dump folder will be created in the current directory and in turn it will contain a folder with the name of the database and inside it will be the bson of the collection + a file containing the metadata
dump/database/collection.metadata.json
dump/database/collection.bson
  1. Finally we go to restore:
mongorestore dump/
  1. To verify that the index has been imported, we can run the following command:
db.collection.getIndexes()

I hope it helps to solve the problem!

Best Regards

6

you forgot to add this detail. please edit your question and add it.

7

Hi @Fabio_Ramohitaj and @Yilmaz_Durmaz ,

Here are the steps which i follow for taking Dump and Restore the MongoDB from My Prod to Stage environment.

Dump From Prod::
nohup /share/infa/AdminScripts/infa/Mongo_DB_Refresh/mongotools/mongodb-database-tools-rhel70-x86_64-100.8.0/bin/mongodump mongodb+srv://:@datahub-prod-g-pl-0.ldvl.mongodb.net/ --db=PTFPIMSPS --numParallelCollections=391 --gzip --out=/share/infa/Backup/mongo_export/PTFPIMSPS_SG_${START_DATE}/ >> /share/infa/Backup/mongo_export/PTFPIMSPS_SG_log_${START_DATE}/PTFPIMSPS_SG_${START_DATE}.log &

It now generates both collection.bson and collection.metadata.json file

Restore in Stage::

In the restore part, when i execute the below command I facing the following issue’s:

nohup /share/infa/AdminScripts/infa/Mongo_DB_Refresh/mongotools/mongodb-database-tools-rhel70-x86_64-100.8.0/bin/mongorestore mongodb+srv://:@datahub-stage-g-pl-0.a94l.mongodb.net/?appName=FF_STFPIMSQS_${BATCHDATE} --nsFrom PTFPIMSPS.* --nsTo STFPIMSQS.* --numParallelCollections=100 --numInsertionWorkersPerCollection=10 --gzip --dir /share/infa/Backup/mongo_export/${PROD_DB_Name}${Identifier}${BATCHDATE}/ --drop >> /share/infa/Backup/mongo_export/${PROD_DB_Name}${Identifier}log${BATCHDATE}/${DB_Name}${Identifier}_${BATCHDATE}.log &

Error:

Failed: STFPIMSQS.SSE036: error creating collection STFPIMSQS.SSE036: error running create command:
(NamespaceExists) Collection STFPIMSQS.SSE036 already exists.

Can you please help me on this.

Regards,
Dinesh S

8

Hi @Dinesh_Reddy_Sanikommu,

Please, format the commands for a better readability.

Formatting code and log snippets in posts - About the Community / Getting Started - MongoDB Developer Community Forums

Can you paste the dbs that are present in the target cluster?
The restore command looks like ok.

Best Regards

9

Sure @Fabio_Ramohitaj , I taking the Dump from PTFPIMSPS database which is in Production and Restoring it in to STFPIMSQS db.

mongorestore mongodb+srv://:@datahub-stage-g-pl-0.a94l.mongodb.net/?appName=FF_STFPIMSQS_${BATCHDATE} --nsFrom PTFPIMSPS.* --nsTo STFPIMSQS.* --numParallelCollections=100 --numInsertionWorkersPerCollection=10 --gzip --dir /share/infa/Backup/mongo_export/${PROD_DB_Name}${Identifier}${BATCHDATE}/ --drop >> /share/infa/Backup/mongo_export/${PROD_DB_Name}${Identifier}log${BATCHDATE}/${DB_Name}* ${Identifier}_${BATCHDATE}.log &

Please let me know if any required.

Also I have the following error while the Index restoration and remaining Indexes are not created.

Failed: STFPIMSQS.SFF004: error creating indexes for STFPIMSQS.SFF004: createIndex error: (DuplicateKey)
Index build failed: c3565a46-31ed-4271-b67d-cbed6f5db828: Collection STFPIMSQS.SFF004 ( b0257688-2d84-40c5-bd60-43d0c851acac )
:: caused by :: E11000 duplicate key error collection: STFPIMSQS.SFF004 index: SFF004_RBA_1 dup key: { SFF004_RBA: 187214632 }

10

Hey @Dinesh_Reddy_Sanikommu,
You should format the code and show me what databases are in the target cluster.

Regards

11

it seems you are trying to restore to an already existing collection/database/index, and --drop does not drop them properly.

a longer error log would be better to help you, as you mentioned in your first post but forgot to attach. please add one.

and for code segments and logs lines, select them and click on the "preformatted text button. or place them between backtick quotes.

1 Like
12

Hi @Fabio_Ramohitaj ,

Here are the list of databases in my stage (target) cluster.

STFPIMSQS, STFPIMSSS, admin, config, local and featureflags

Regards

13

Hi @Dinesh_Reddy_Sanikommu
The --drop option is ineffective when the nsFrom and nsTo options are used, I hypothesize because maybe an attempt is made to drop the nsFrom db (which does not exist) or something similar (you would have to add verbosity to verify that this is effectively the case), so to solve the problem, you will have to manually drop the STFPIMSQS database into the target cluster in order for the restore to be executed correctly.
If it is necessary, make a backup of the database that you will have to drop before the drop.

Regards

1 Like
14

Hi @Fabio_Ramohitaj , By manually dropping the STFPIMSQS database would drop all the collections, where in the above process of the restore command which I have shared will only restore the collections existing from Prod to Stage, and will not impact the rest of the stage collections(Not exist in Prod). Could you just any other feasible way’s.

15

using --drop you are already telling the command to drop the collections before restoring from backup, which means you are already impacting them.

please refer to official documentation:
mongorestore - MongoDB Database Tools

1 Like