Fatal Assertion 40292 at src/mongo/db/repl/replication_recovery.cpp

Hi All,

We are running MongoDB on Docker swarm and it was working fine till yesterday. We have stopped the docker daemon for patching post that we are getting below error.

I ran mongodb repair and now am getting below error. Can someone please suggest on this?

Oplog entry at { :Timestamp(1624030066,1)) is missing; actual entry found is { : Timestamp(16246-83,3}}
Fatal Assertion 40292 at src/mongo/db/repl/replication_Recover.cpp 220

Welcome to the MongoDB Community Forums @Siva_Reddy_Kotigari !

The error message indicates a problem reading your WiredTiger data files; perhaps this mongod was incorrectly shutdown during your patching process.

Since this mongod has an oplog and is presumably part of a replica set, you should Resync from a healthy member of the same replica set rather than running repair.

The repair process salvages data that can be successfully read (ignoring data that cannot) and should only be used on a standalone mongod when you have no other options. Repairing a replica set member with data file issues may result in a data set which is not consistent with other members of the replica set, which in turn will lead to further operational challenges.

Copying data from another replica set member or a recent backup is a much more reliable option to return your deployment to a working state.

Regards,
Stennie

Hi @Stennie,

Thanks for the replay.

We are running 3 mongo services, now i have copied the files from mongodb2 to mongodb1 and started the services. I can see all services are up and running but in logs am seeing connection refused. Please suggested

mongodb01

2021-06-22T12:51:22.910+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to mongodb_db03:27019
2021-06-22T12:51:22.911+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb_db03:27019 - HostUnreachable: Connection refused
2021-06-22T12:51:22.911+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb_db03:27019 due to failed operation on a connection
2021-06-22T12:51:22.911+0000 I REPL_HB [replexec-5] Error in heartbeat (requestId: 2024) to mongodb_db03:27019, response status: HostUnreachable: Connection refused

mongodb02

2021-06-22T12:51:22.910+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Connecting to mongodb_db03:27019
2021-06-22T12:51:22.911+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb_db03:27019 - HostUnreachable: Connection refused
2021-06-22T12:51:22.911+0000 I ASIO [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb_db03:27019 due to failed operation on a connection
2021-06-22T12:51:22.911+0000 I REPL_HB [replexec-5] Error in heartbeat (requestId: 2024) to mongodb_db03:27019, response status: HostUnreachable: Connection refused

mongodb03

I CONTROL [initandlisten] options: { config: “/etc/mongod.conf”, net: { bindIpAll: true, port: 27019, ssl: { CAFile: “/etc/certs/ca.pem”, PEMKeyFile: “/etc/certs/cert.pem”, allowConnectionsWithoutCertificates: true, allowInvalidHostnames: true, mode: “preferSSL” } }, replication: { oplogSizeMB: 400, replSetName: “graylog” } }
W - [initandlisten] Detected unclean shutdown - /data/db/mongod.lock is not empty.
I - [initandlisten] Detected data files in /data/db created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=63873M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="