Recover MongoDB Data from Physical Storage

Ops and Admin
1

Hello,

I have been trying to recover my data store from the physical location. I was running MongoDB Community Version 4.0.10 and accidentally deleted the /bin directory. However, my MongoDB data was located at /var/lib/mongodb and had 105 files in it. According to the log generated while deletion, I got the confirmation that none of the files of the data directory was deleted.

So I copied the entire data from “/var/lib/mongodb” to my local machine and kept it in the same location, installed the same version of MongoDB, which is 4.0.10, but could not manage to retrieve the data.

After that, I have run the following command to see if I can repair the data:

sudo mongod --repair --dbpath /var/lib/mongodb

This process also did not help as I received the message “Aborted” affer I executed it. In the log file of MongoDB located at /etc/mongod.log, I have found a line that says:

…
read checksum error for 4096B block at offset 249856: block header checksum of 0x1c71aba3 doesn’t match expected checksum of 0x5881888d"}}
…

I am sure that none of the files in the server data directory has been deleted.

I have also used the following commands to give permissions to my local data directory:

sudo chown -R mongodb:mongodb /var/lib/mongodb 
sudo chown mongodb:mongodb /tmp/mongodb-27017.sock

But I still could not recover the data. Can someone please help?

2

Even though the binary file was deleted it is quite likely that mongod was still running. Copy the files while mongod is running would result in the files not matching/being correct as you have experienced.

If the original system is still running, you can check for the running mongod process pgrep mongod and kill it with pkill mongod then copying the file should be successful.

3

Unfortunately, the original system is not running. So, can’t do this anymore. Is there any other way?

4

Short answer: no.

Longer answer:

1 Like
5

So, this is how to problem was solved.

Step 1:
We have installed the same version of MongoDB that was used in the server. For our case, the version was 4.0.10. We have used the following command to download the specific version:

sudo apt-get install -y mongodb-org=4.0.10 mongodb-org-server=4.0.10 mongodb-org-shell=4.0.10 mongodb-org-mongos=4.0.10 mongodb-org-tools=4.0.10

Step 2:
The version installed at step 1 was storing it’s data on /var/lib/mongodb. So, we started the MongoDB server using

mongod --dbpath /var/lib/mongodb

Step 3:
We have started the mongo terminal client using the command:

mongo

Step 4:
After entering into the client, we have created a new database using the command

use new_database

Step 5:
We have created a dummy collection inside our “new_database” using the command:

db.newCollection.insert({hi: "helo"})

Step 6:
We have checked the stats of our new collection using:

db.newCollectoin.stats()

Step 7:
From the output file generated at step 6, we have looked for the keyword “uri” to see under which .wt file, our “newCollection” collection is stored. The uri key in the output should look something like this:

		"uri" : "statistics:table:collection-44--4912736618580810110",

Step 8:
We have kept a local copy of our server database at a location, let’s call it /home/user/old_database. Inside it, let’s say, there’s a .wt file called “collection-12–4912736618580810442.wt”. We copied this .wt file to our new .wt file that represents the “newCollection” collection. We used the command:

sudo cp /home/user/old_database/collection-12--4912736618580810442.wt /var/lib/collection-44--4912736618580810110.wt

Our intention is to make the new database consider the server .wt file as a distorted version of new databases’ .wt file.

Step 9:
We stopped the MongoDB client and server and run the repair command on our new database:

sudo mongod --dbpath /var/lib/mongodb --repair

Step 10:
After a successful repair, we have seen the exit code 0, as output of our command in line 9. So, we restarted the MongoDB server again, just like step 2:

mongod --dbpath /var/lib/mongodb

Step 11:
We started the mongo client again, choose the new_database again and looked at the collection like this:

db.newCollection.find({}).pretty()

Fortunately, we could see the old documents stored in one of the collections of our old_database.

Step 12:
We repeated steps 1 to 11 again for other .wt files in the old_database.

We have learned this technique from: Repairing MongoDB When WiredTiger.wt File is Corrupted | by Ido Ozeri | Medium

Hope this write-up will help some other teams in the future. Please don’t forget to keep regular backup as well :smiley:

1 Like
6

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.