- Administration >
- Administration Tutorials >
- Backup and Recovery >
- Recover Data after an Unexpected Shutdown
Recover Data after an Unexpected Shutdown¶
On this page
If MongoDB does not shutdown cleanly [1] the on-disk representation of the data files will likely reflect an inconsistent state which could lead to data corruption. [2]
To prevent data inconsistency and corruption, always shut down the database cleanly and use the durability journaling. MongoDB writes data to the journal, by default, every 100 milliseconds, such that MongoDB can always recover to a consistent state even in the case of an unclean shutdown due to power loss or other system failure.
If you are not running as part of a replica set and do
not have journaling enabled, use the following procedure to recover
data that may be in an inconsistent state. If you are running as part
of a replica set, you should always restore from a backup or restart
the mongod
instance with an empty dbPath
and
allow MongoDB to perform an initial sync to restore the data.
See also
The Administration documents, including
Replica Set Syncing, and the
documentation on the --repair
repairPath
and
storage.journal.enabled
settings.
[1] | To ensure a clean shut down, use the
db.shutdownServer() from the mongo shell, your
control script, the mongod --shutdown option on Linux
systems, “Control-C” when running mongod in interactive
mode, or kill $(pidof mongod) or kill -2 $(pidof mongod) . |
[2] | You can also use the db.collection.validate()
method to test the integrity of a single collection. However, this
process is time consuming, and without journaling you can safely
assume that the data is in an invalid state and you should either
run the repair operation or resync from an intact member of the
replica set. |
Process¶
Indications¶
When you are aware of a mongod
instance running without
journaling that stops unexpectedly and you’re not running with
replication, you should always run the repair operation before
starting MongoDB again. If you’re using replication, then restore from
a backup and allow replication to perform an initial sync to restore data.
If the mongod.lock
file in the data directory specified by
dbPath
, /data/db
by default, is not a zero-byte file,
then mongod
will refuse to start, and you will find a
message that contains the following line in your MongoDB log our
output:
This indicates that you need to run mongod
with the
--repair
option. If you run repair when
the mongodb.lock
file exists in your dbPath
, or the
optional --repairpath
, you will see a
message that contains the following line:
If you see this message, as a last resort you may remove the lockfile and run the repair operation before starting the database normally, as in the following procedure:
Overview¶
Warning
Recovering a member of a replica set.
Do not use this procedure to recover a member of a replica set. Instead you should either restore from a backup or perform an initial sync using data from an intact member of the set, as described in Resync a Member of a Replica Set.
There are two processes to repair data files that result from an unexpected shutdown:
Use the
--repair
option in conjunction with the--repairpath
option.mongod
will read the existing data files, and write the existing data to new data files.You do not need to remove the
mongod.lock
file before using this procedure.Use the
--repair
option.mongod
will read the existing data files, write the existing data to new files and replace the existing, possibly corrupt, files with new files.You must remove the
mongod.lock
file before using this procedure.
Note
--repair
functionality is also
available in the shell with the db.repairDatabase()
helper for the repairDatabase
command.
Procedures¶
Important
Always Run mongod
as the same user to avoid
changing the permissions of the MongoDB data files.
Repair Data Files and Preserve Original Files¶
To repair your data files using the --repairpath
option to preserve the original data files unmodified.
Repair Data Files without Preserving Original Files¶
To repair your data files without preserving the original files, do
not use the --repairpath
option, as in
the following procedure:
Warning
After you remove the mongod.lock
file you must run the
--repair
process before using your
database.
Start mongod
using the option to replace the original files with the repaired files.¶
Start the mongod
instance using the --repair
option and the --repairpath
option. Issue a command similar to the
following:
When this completes, the new repaired data files will be in the
/data/db0
directory.
Start mongod
with the new data directory.¶
Start mongod
using the following invocation to point the
dbPath
at /data/db0
:
Once you confirm that the data files are operational you may delete or
archive the old data files in the /data/db
directory. You may also
wish to move the repaired files to the old database location or update
the dbPath
to indicate the new location.
Start mongod
using the option to replace the original files with the repaired files.¶
Start the mongod
instance using the --repair
option, which replaces the original data files with
the repaired data files. Issue a command similar to the following:
When this completes, the repaired data files will replace the original
data files in the /data/db
directory.
mongod.lock
¶
In normal operation, you should never remove the mongod.lock
file and start mongod
. Instead consider the one of the above methods
to recover the database and remove the lock files. In dire
situations you can remove the lockfile, and start the database using the
possibly corrupt files, and attempt to recover data from the database;
however, it’s impossible to predict the state of the database in these
situations.
If you are not running with journaling, and your database shuts down unexpectedly for any reason, you should always proceed as if your database is in an inconsistent and likely corrupt state. If at all possible restore from backup or, if running as a replica set, restore by performing an initial sync using data from an intact member of the set, as described in Resync a Member of a Replica Set.