Hi,
I hope you are doing well. I have an issue if anyone can help please. Actually we have a mongoDB 3.6 running in standalone, something got wrong and we lost the metadata of the DB, now all we got is a directory data with all the colle--.wt and idex-***.wt files. Is there any solution to retrieve the data from those wt files and import them in a new mongoDB instance.
We would be so gratefull if someone can help.
Have a nice day.
Best regards,
Aissa
Just to be sure there is only collection-*.wt and index-*.wt remaining ?
Hi, thank you for your reply. Yes only the cool and index et files the other file containing the metadata _md_catalog are lost. Thank you again for your interest.
I recall seeing a post some time back about using wt to dump the collections from the .wt files.
It won’t put it all back together but at least you have the data. I’ll see if I can find it. It did need compiling as the OS and release versions of wt didn’t include snappy compression.
Hi @Aissa_Elaffani,
This problem makes me extremely curious @chris & @Aissa_Elaffani 
I tried to simulate your situation in a small way (Unfortunately, in this quick simulation, I am using the 4.4), can you confirm that it is in the following way:
[root@mongodbstandalone mongo]# ls -altrh
totale 488K
drwxr-xr-x 4 mongod mongod 35 2 nov 05.47 ..
-rw------- 1 mongod mongod 21 2 nov 05.47 WiredTiger.lock
-rw------- 1 mongod mongod 50 2 nov 05.47 WiredTiger
-rw------- 1 mongod mongod 114 2 nov 05.47 storage.bson
-rw------- 1 mongod mongod 20K 2 nov 06.24 index-8--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 2 nov 06.24 collection-7--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 2 nov 14.48 index-10--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 2 nov 14.48 collection-9--5791277203608009656.wt
-rw------- 1 mongod mongod 12K 2 nov 15.19 index-5--5791277203608009656.wt
-rw------- 1 mongod mongod 12K 2 nov 15.19 collection-4--5791277203608009656.wt
-rw------- 1 mongod mongod 4,0K 9 nov 22.37 WiredTigerHS.wt
-rw------- 1 mongod mongod 36K 9 nov 22.37 sizeStorer.wt
-rw------- 1 mongod mongod 36K 9 nov 22.37 _mdb_catalog.wt
-rw------- 1 mongod mongod 5 9 nov 22.37 mongod.lock
-rw------- 1 mongod mongod 20K 9 nov 22.37 index-1--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 9 nov 22.37 collection-0--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 9 nov 22.38 index-3--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 9 nov 22.38 collection-2--5791277203608009656.wt
-rw------- 1 mongod mongod 84K 9 nov 22.38 WiredTiger.wt
-rw------- 1 mongod mongod 1,5K 9 nov 22.38 WiredTiger.turtle
drwxr-xr-x 4 mongod mongod 4,0K 9 nov 22.38 .
drwx------ 2 mongod mongod 110 9 nov 22.38 journal
-rw------- 1 mongod mongod 12K 9 nov 22.38 index-6--5791277203608009656.wt
drwx------ 2 mongod mongod 113 9 nov 22.39 diagnostic.data
[root@mongodbstandalone mongo]# rm -rf WiredTiger.lock WiredTiger storage.bson mongod.lock journal WiredTiger.turtle diagnostic.data
[root@mongodbstandalone mongo]# ls -altrh
totale 424K
drwxr-xr-x 4 mongod mongod 35 2 nov 05.47 ..
-rw------- 1 mongod mongod 20K 2 nov 06.24 index-8--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 2 nov 06.24 collection-7--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 2 nov 14.48 index-10--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 2 nov 14.48 collection-9--5791277203608009656.wt
-rw------- 1 mongod mongod 12K 2 nov 15.19 index-5--5791277203608009656.wt
-rw------- 1 mongod mongod 12K 2 nov 15.19 collection-4--5791277203608009656.wt
-rw------- 1 mongod mongod 4,0K 9 nov 22.37 WiredTigerHS.wt
-rw------- 1 mongod mongod 36K 9 nov 22.37 _mdb_catalog.wt
-rw------- 1 mongod mongod 20K 9 nov 22.37 index-1--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 9 nov 22.37 collection-0--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 9 nov 22.38 index-3--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 9 nov 22.38 collection-2--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 9 nov 22.39 sizeStorer.wt
-rw------- 1 mongod mongod 12K 9 nov 22.40 index-6--5791277203608009656.wt
-rw------- 1 mongod mongod 84K 9 nov 22.40 WiredTiger.wt
drwxr-xr-x 2 mongod mongod 4,0K 9 nov 22.41 .
[root@mongodbstandalone mongo]# systemctl status mongod
● mongod.service - MongoDB Database Server
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since gio 2023-11-09 22:41:06 CET; 16s ago
Docs: https://docs.mongodb.org/manual
Process: 1026 ExecStart=/usr/bin/mongod $OPTIONS (code=killed, signal=ABRT)
Main PID: 1026 (code=killed, signal=ABRT)
nov 09 22:37:56 mongodbstandalone.test.com systemd[1]: Started MongoDB Database Server.
nov 09 22:37:56 mongodbstandalone.test.com mongod[1026]: {"t":{"$date":"2023-11-09T21:37:56.698Z"},"s":"I", "c":"CONTROL", "id":7484500, "ctx":"main","msg":"Environment variable MONGO..." to false"}
nov 09 22:41:06 mongodbstandalone.test.com systemd[1]: mongod.service: main process exited, code=killed, status=6/ABRT
nov 09 22:41:06 mongodbstandalone.test.com systemd[1]: Unit mongod.service entered failed state.
nov 09 22:41:06 mongodbstandalone.test.com systemd[1]: mongod.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
Best Regards
Thank you Chris again for your interest.
as I said Fabio all what i got right now is the collections and index wt files. but the other files like _mdb_catalog.wt got lost. so right now i am trying to find a solution to restore the data from the wt files without the metada files. idk even if it is possible or not. Thank you for reply.
I don’t have a tidy solution for you but I’m working on building wt 11.1.0 to see if that will help.
The post I’m thinking of was @alexbevi’s https://www.alexbevi.com/blog/2020/01/23/troubleshooting-and-fixing-invariant-failure-featuretracker/
Hi @Aissa_Elaffani,
In my lab, i’ ve simply restarted the instance and the service is back up.
Regards
@Fabio_Ramohitaj Did it actually have existing data or did you end up with a shiny new empty database ?
Hi @chris 
Tomorrow in the morning I will pull up the vm again and post the result.
Best Regards
Hi @chris,
Here is the initial situation:
[root@mongodbstandalone mongo]# ls -lathr
totale 412K
drwxr-xr-x 4 mongod mongod 35 2 nov 05.47 ..
-rw------- 1 mongod mongod 21 2 nov 05.47 WiredTiger.lock
-rw------- 1 mongod mongod 50 2 nov 05.47 WiredTiger
-rw------- 1 mongod mongod 114 2 nov 05.47 storage.bson
-rw------- 1 mongod mongod 20K 2 nov 06.24 index-8--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 2 nov 06.24 collection-7--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 2 nov 14.48 index-10--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 2 nov 14.48 collection-9--5791277203608009656.wt
-rw------- 1 mongod mongod 12K 2 nov 15.19 index-5--5791277203608009656.wt
-rw------- 1 mongod mongod 12K 2 nov 15.19 collection-4--5791277203608009656.wt
-rw------- 1 mongod mongod 12K 2 nov 15.32 index-6--5791277203608009656.wt
-rw------- 1 mongod mongod 4,0K 11 nov 11.16 WiredTigerHS.wt
-rw------- 1 mongod mongod 84K 11 nov 11.16 WiredTiger.wt
-rw------- 1 mongod mongod 1,5K 11 nov 11.16 WiredTiger.turtle
drwxr-xr-x 4 mongod mongod 4,0K 11 nov 11.16 .
-rw------- 1 mongod mongod 36K 11 nov 11.16 sizeStorer.wt
-rw------- 1 mongod mongod 36K 11 nov 11.16 _mdb_catalog.wt
-rw------- 1 mongod mongod 5 11 nov 11.16 mongod.lock
drwx------ 2 mongod mongod 142 11 nov 11.16 journal
-rw------- 1 mongod mongod 20K 11 nov 11.16 index-1--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 11 nov 11.16 collection-0--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 11 nov 11.16 collection-2--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 11 nov 11.16 index-3--5791277203608009656.wt
drwx------ 2 mongod mongod 113 11 nov 11.17 diagnostic.data
[root@mongodbstandalone mongo]# systemctl status mongod
● mongod.service - MongoDB Database Server
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: active (running) since sab 2023-11-11 11:16:52 CET; 39s ago
Docs: https://docs.mongodb.org/manual
Main PID: 1032 (mongod)
CGroup: /system.slice/mongod.service
└─1032 /usr/bin/mongod -f /etc/mongod.conf
nov 11 11:16:52 mongodbstandalone.test.com systemd[1]: Started MongoDB Database Server.
nov 11 11:16:53 mongodbstandalone.test.com mongod[1032]: {"t":{"$date":"2023-11-11T10:16:53.275Z"},"s":"I", "c":"CONTROL", "id":7484500, "ctx":"main","msg":"Environment variable MONGO..." to false"}
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongodbstandalone mongo]# mongo
MongoDB shell version v4.4.25
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("ff2cfc5e-b73c-4f37-9afa-59b44281243e") }
MongoDB server version: 4.4.25
---
The server generated these startup warnings when booting:
2023-11-11T11:16:54.180+01:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
2023-11-11T11:16:54.181+01:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
2023-11-11T11:16:54.181+01:00: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. We suggest setting it to 'never'
---
MongoDB Enterprise > show dbs
admin 0.000GB
config 0.000GB
formigration 0.000GB
local 0.000GB
personalInformation 0.000GB
and here is the situation after deleting the non .wt files
[root@mongodbstandalone mongo]# rm -rf WiredTiger.lock WiredTiger storage.bson mongod.lock journal WiredTiger.turtle diagnostic.data
[root@mongodbstandalone mongo]# ls -latrh
totale 424K
drwxr-xr-x 4 mongod mongod 35 2 nov 05.47 ..
-rw------- 1 mongod mongod 20K 2 nov 06.24 index-8--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 2 nov 06.24 collection-7--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 2 nov 14.48 index-10--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 2 nov 14.48 collection-9--5791277203608009656.wt
-rw------- 1 mongod mongod 12K 2 nov 15.19 index-5--5791277203608009656.wt
-rw------- 1 mongod mongod 12K 2 nov 15.19 collection-4--5791277203608009656.wt
-rw------- 1 mongod mongod 4,0K 11 nov 11.16 WiredTigerHS.wt
-rw------- 1 mongod mongod 36K 11 nov 11.16 _mdb_catalog.wt
-rw------- 1 mongod mongod 20K 11 nov 11.16 index-1--5791277203608009656.wt
-rw------- 1 mongod mongod 20K 11 nov 11.16 collection-0--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 11 nov 11.17 index-3--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 11 nov 11.17 collection-2--5791277203608009656.wt
-rw------- 1 mongod mongod 36K 11 nov 11.18 sizeStorer.wt
-rw------- 1 mongod mongod 12K 11 nov 11.20 index-6--5791277203608009656.wt
-rw------- 1 mongod mongod 84K 11 nov 11.20 WiredTiger.wt
drwxr-xr-x 2 mongod mongod 4,0K 11 nov 11.21 .
[root@mongodbstandalone mongo]# systemctl status mongod
● mongod.service - MongoDB Database Server
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: failed (Result: signal) since sab 2023-11-11 11:21:28 CET; 14s ago
Docs: https://docs.mongodb.org/manual
Process: 1032 ExecStart=/usr/bin/mongod $OPTIONS (code=killed, signal=ABRT)
Main PID: 1032 (code=killed, signal=ABRT)
nov 11 11:16:52 mongodbstandalone.test.com systemd[1]: Started MongoDB Database Server.
nov 11 11:16:53 mongodbstandalone.test.com mongod[1032]: {"t":{"$date":"2023-11-11T10:16:53.275Z"},"s":"I", "c":"CONTROL", "id":7484500, "ctx":"main","msg":"Environment variable MONGO..." to false"}
nov 11 11:21:28 mongodbstandalone.test.com systemd[1]: mongod.service: main process exited, code=killed, status=6/ABRT
nov 11 11:21:28 mongodbstandalone.test.com systemd[1]: Unit mongod.service entered failed state.
nov 11 11:21:28 mongodbstandalone.test.com systemd[1]: mongod.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongodbstandalone ~]# systemctl restart mongod
[root@mongodbstandalone ~]# systemctl status mongod
● mongod.service - MongoDB Database Server
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: active (running) since sab 2023-11-11 12:51:28 CET; 1h 1min ago
Docs: https://docs.mongodb.org/manual
Main PID: 1027 (mongod)
CGroup: /system.slice/mongod.service
└─1027 /usr/bin/mongod -f /etc/mongod.conf
nov 11 12:51:28 mongodbstandalone.test.com systemd[1]: Started MongoDB Database Server.
nov 11 12:51:28 mongodbstandalone.test.com mongod[1027]: {"t":{"$date":"2023-11-11T11:51:28.897Z"},"s":"I", "c":"CONTROL", "id":7484500, "ctx":"main","msg":"Environment variable MONGO..." to false"}
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongodbstandalone mongo]# mongo
MongoDB shell version v4.4.25
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("0a53271b-e867-4546-b9ce-ea8303ba78c9") }
MongoDB server version: 4.4.25
---
The server generated these startup warnings when booting:
2023-11-11T11:21:54.570+01:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
2023-11-11T11:21:54.571+01:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
2023-11-11T11:21:54.571+01:00: /sys/kernel/mm/transparent_hugepage/defrag is 'always'. We suggest setting it to 'never'
---
MongoDB Enterprise > show dbs
admin 0.000GB
config 0.000GB
local 0.000GB
MongoDB Enterprise >
there is nothing left
too clean 
I’ ve try to reproduce that solution to this case:
but it doesn’t work for me
It didnt work for me neither :s