6.0.3 update 6.0.5 issue

We are on RHEL8, the update of 6.0.2 to 6.0.5 using rpm seems to create issue on startup of mongod process

we start the update on one of our standby node and high level steps as below.

  1. stop mongod process on standby node using “systemctl stop mongod”
  2. yum update mongodb*
  3. check rpm package goes to 6.0.5
  4. bring up mongod on standby node using “systemctl start mongod”
  5. check mongod process status using “systemctl status mongod”

the process cannot startup with mongod.log errors like below

{"t":{"$date":"2023-05-11T11:52:27.789+00:00"},"s":"I",  "c":"REPL",     "id":4280505, "ctx":"initandlisten","msg":"Creating any necessary TenantMigrationAccessBlockers for unfinished migrations"}
{"t":{"$date":"2023-05-11T11:52:27.800+00:00"},"s":"I",  "c":"REPL",     "id":4280506, "ctx":"initandlisten","msg":"Reconstructing prepared transactions"}
{"t":{"$date":"2023-05-11T11:52:27.801+00:00"},"s":"I",  "c":"REPL",     "id":4280507, "ctx":"initandlisten","msg":"Loaded replica set config, scheduled callback to set local config"}
{"t":{"$date":"2023-05-11T11:52:27.801+00:00"},"s":"I",  "c":"REPL",     "id":4280508, "ctx":"ReplCoord-0","msg":"Attempting to set local replica set config; validating config for startup"}
{"t":{"$date":"2023-05-11T11:52:27.801+00:00"},"s":"I",  "c":"STORAGE",  "id":22262,   "ctx":"initandlisten","msg":"Timestamp monitor starting"}
{"t":{"$date":"2023-05-11T11:52:27.802+00:00"},"s":"I",  "c":"CONTROL",  "id":20714,   "ctx":"LogicalSessionCacheRefresh","msg":"Failed to refresh session cache, will try again at the next refresh interval","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}
{"t":{"$date":"2023-05-11T11:52:27.803+00:00"},"s":"I",  "c":"CONTROL",  "id":20711,   "ctx":"LogicalSessionCacheReap","msg":"Failed to reap transaction table","attr":{"error":"NotYetInitialized: Replication has not yet been configured"}}
{"t":{"$date":"2023-05-11T11:52:27.803+00:00"},"s":"I",  "c":"REPL",     "id":40440,   "ctx":"initandlisten","msg":"Starting the TopologyVersionObserver"}
{"t":{"$date":"2023-05-11T11:52:27.803+00:00"},"s":"I",  "c":"REPL",     "id":40445,   "ctx":"TopologyVersionObserver","msg":"Started TopologyVersionObserver"}
{"t":{"$date":"2023-05-11T11:52:27.804+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
{"t":{"$date":"2023-05-11T11:52:27.804+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"10.58.156.15"}}
{"t":{"$date":"2023-05-11T11:52:27.804+00:00"},"s":"I",  "c":"NETWORK",  "id":23015,   "ctx":"listener","msg":"Listening on","attr":{"address":"127.0.0.1"}}
{"t":{"$date":"2023-05-11T11:52:27.804+00:00"},"s":"I",  "c":"NETWORK",  "id":23016,   "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
{"t":{"$date":"2023-05-11T11:52:27.806+00:00"},"s":"I",  "c":"CONTROL",  "id":23377,   "ctx":"SignalHandler","msg":"Received signal","attr":{"signal":15,"error":"Terminated"}}
{"t":{"$date":"2023-05-11T11:52:27.806+00:00"},"s":"I",  "c":"CONTROL",  "id":23378,   "ctx":"SignalHandler","msg":"Signal was sent by kill(2)","attr":{"pid":1,"uid":0}}
{"t":{"$date":"2023-05-11T11:52:27.806+00:00"},"s":"I",  "c":"CONTROL",  "id":23381,   "ctx":"SignalHandler","msg":"will terminate after current cmd ends"}
{"t":{"$date":"2023-05-11T11:52:27.806+00:00"},"s":"I",  "c":"REPL",     "id":4784900, "ctx":"SignalHandler","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":15000}}
{"t":{"$date":"2023-05-11T11:52:27.806+00:00"},"s":"I",  "c":"REPL",     "id":4794602, "ctx":"SignalHandler","msg":"Attempting to enter quiesce mode"}
{"t":{"$date":"2023-05-11T11:52:27.806+00:00"},"s":"I",  "c":"-",        "id":6371601, "ctx":"SignalHandler","msg":"Shutting down the FLE Crud thread pool"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"FLECrudNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"COMMAND",  "id":4784901, "ctx":"SignalHandler","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"REPL",     "id":40441,   "ctx":"SignalHandler","msg":"Stopping TopologyVersionObserver"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"REPL",     "id":40447,   "ctx":"TopologyVersionObserver","msg":"Stopped TopologyVersionObserver"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"SHARDING", "id":4784902, "ctx":"SignalHandler","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"CONTROL",  "id":4784903, "ctx":"SignalHandler","msg":"Shutting down the LogicalSessionCache"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"NETWORK",  "id":20562,   "ctx":"SignalHandler","msg":"Shutdown: going to close listening sockets"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"NETWORK",  "id":23017,   "ctx":"listener","msg":"removing socket file","attr":{"path":"/tmp/mongodb-27017.sock"}}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"NETWORK",  "id":4784905, "ctx":"SignalHandler","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"CONTROL",  "id":4784906, "ctx":"SignalHandler","msg":"Shutting down the FlowControlTicketholder"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"-",        "id":20520,   "ctx":"SignalHandler","msg":"Stopping further Flow Control ticket acquisitions."}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"REPL",     "id":4784907, "ctx":"SignalHandler","msg":"Shutting down the replica set node executor"}
{"t":{"$date":"2023-05-11T11:52:27.807+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"ReplNodeDbWorkerNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"CONTROL",  "id":4784908, "ctx":"SignalHandler","msg":"Shutting down the PeriodicThreadToAbortExpiredTransactions"}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"REPL",     "id":4784909, "ctx":"SignalHandler","msg":"Shutting down the ReplicationCoordinator"}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"REPL",     "id":5074000, "ctx":"SignalHandler","msg":"Shutting down the replica set aware services."}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"REPL",     "id":5123006, "ctx":"SignalHandler","msg":"Shutting down PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","numInstances":0,"numOperationContexts":0}}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"TenantMigrationDonorServiceNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"REPL",     "id":5123006, "ctx":"SignalHandler","msg":"Shutting down PrimaryOnlyService","attr":{"service":"ShardSplitDonorService","numInstances":0,"numOperationContexts":0}}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"ShardSplitDonorServiceNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"REPL",     "id":5123006, "ctx":"SignalHandler","msg":"Shutting down PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","numInstances":0,"numOperationContexts":0}}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"TenantMigrationRecipientServiceNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2023-05-11T11:52:27.808+00:00"},"s":"I",  "c":"REPL",     "id":21328,   "ctx":"SignalHandler","msg":"Shutting down replication subsystems"}
{"t":{"$date":"2023-05-11T11:52:27.987+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":400}}
{"t":{"$date":"2023-05-11T11:52:28.387+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":600}}
{"t":{"$date":"2023-05-11T11:52:28.988+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":800}}
{"t":{"$date":"2023-05-11T11:52:29.789+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":1000}}
{"t":{"$date":"2023-05-11T11:52:30.790+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":1200}}
{"t":{"$date":"2023-05-11T11:52:31.992+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":1400}}
{"t":{"$date":"2023-05-11T11:52:33.394+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":1600}}
{"t":{"$date":"2023-05-11T11:52:34.996+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":1800}}
{"t":{"$date":"2023-05-11T11:52:36.799+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":2000}}
{"t":{"$date":"2023-05-11T11:52:38.801+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":2200}}
{"t":{"$date":"2023-05-11T11:52:41.004+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":2400}}
{"t":{"$date":"2023-05-11T11:52:43.407+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":2600}}
{"t":{"$date":"2023-05-11T11:52:46.010+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":2800}}
{"t":{"$date":"2023-05-11T11:52:48.813+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":3000}}
{"t":{"$date":"2023-05-11T11:52:51.817+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":3200}}
{"t":{"$date":"2023-05-11T11:52:55.019+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":3400}}
{"t":{"$date":"2023-05-11T11:52:58.324+00:00"},"s":"I",  "c":"NETWORK",  "id":20125,   "ctx":"ReplCoord-0","msg":"DBClientConnection failed to receive message","attr":{"connString":"dehamsl5570.int.kn:27017","error":"NetworkTimeout: Socket operation timed out"}}
{"t":{"$date":"2023-05-11T11:52:58.324+00:00"},"s":"I",  "c":"NETWORK",  "id":20117,   "ctx":"ReplCoord-0","msg":"Can't authenticate as internal user","attr":{"connString":"dehamsl5570.int.kn:27017 failed","error":{"code":6,"codeName":"HostUnreachable","errmsg":"network error while attempting to run command 'ismaster' on host 'dehamsl5570.int.kn:27017' "}}}
{"t":{"$date":"2023-05-11T11:52:58.324+00:00"},"s":"I",  "c":"NETWORK",  "id":4834701, "ctx":"ReplCoord-0","msg":"isSelf could not authenticate internal user","attr":{"hostAndPort":"dehamsl5570.int.kn:27017","error":{"code":6,"codeName":"HostUnreachable","errmsg":"network error while attempting to run command 'ismaster' on host 'dehamsl5570.int.kn:27017' "}}}
{"t":{"$date":"2023-05-11T11:52:58.324+00:00"},"s":"W",  "c":"REPL",     "id":21405,   "ctx":"ReplCoord-0","msg":"Locally stored replica set configuration does not have a valid entry for the current node; waiting for reconfig or remote heartbeat","attr":{"error":{"code":74,"codeName":"NodeNotFound","errmsg":"No host described in new configuration with {version: 2, term: 7} for replica set share-dev-sit-rs maps to this node"},"localConfig":{"_id":"share-dev-sit-rs","version":2,"term":7,"members":[{"_id":0,"host":"denotsl3877.int.kn:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"denotsl3878.int.kn:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":0.5,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":2,"host":"dehamsl5570.int.kn:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":0.5,"tags":{},"secondaryDelaySecs":0,"votes":1}],"protocolVersion":1,"writeConcernMajorityJournalDefault":true,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"catchUpTakeoverDelayMillis":30000,"getLastErrorModes":{},"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":{"$oid":"63c57549bf87da334b3f786f"}}}}}
{"t":{"$date":"2023-05-11T11:52:58.324+00:00"},"s":"I",  "c":"REPL",     "id":4280509, "ctx":"ReplCoord-0","msg":"Local configuration validated for startup"}
{"t":{"$date":"2023-05-11T11:52:58.324+00:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"ReplCoord-0","msg":"Setting new configuration state","attr":{"newState":"ConfigSteady","oldState":"ConfigStartingUp"}}
{"t":{"$date":"2023-05-11T11:52:58.325+00:00"},"s":"I",  "c":"REPL",     "id":21392,   "ctx":"ReplCoord-0","msg":"New replica set config in use","attr":{"config":{"_id":"share-dev-sit-rs","version":2,"term":7,"members":[{"_id":0,"host":"denotsl3877.int.kn:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"denotsl3878.int.kn:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":0.5,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":2,"host":"dehamsl5570.int.kn:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":0.5,"tags":{},"secondaryDelaySecs":0,"votes":1}],"protocolVersion":1,"writeConcernMajorityJournalDefault":true,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"catchUpTakeoverDelayMillis":30000,"getLastErrorModes":{},"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":{"$oid":"63c57549bf87da334b3f786f"}}}}}
{"t":{"$date":"2023-05-11T11:52:58.325+00:00"},"s":"I",  "c":"REPL",     "id":21394,   "ctx":"ReplCoord-0","msg":"This node is not a member of the config"}
{"t":{"$date":"2023-05-11T11:52:58.325+00:00"},"s":"I",  "c":"REPL",     "id":21358,   "ctx":"ReplCoord-0","msg":"Replica set state transition","attr":{"newState":"REMOVED","oldState":"STARTUP"}}
{"t":{"$date":"2023-05-11T11:52:58.325+00:00"},"s":"I",  "c":"REPL",     "id":21320,   "ctx":"ReplCoord-0","msg":"Updated term","attr":{"term":7}}
{"t":{"$date":"2023-05-11T11:52:58.325+00:00"},"s":"I",  "c":"REPL",     "id":4280511, "ctx":"ReplCoord-0","msg":"Set local replica set config"}
{"t":{"$date":"2023-05-11T11:52:58.325+00:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ShutdownInProgress: Shutdown in progress","nextWakeupMillis":3600}}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"ReplNetwork","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"SHARDING", "id":4784910, "ctx":"SignalHandler","msg":"Shutting down the ShardingInitializationMongoD"}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"REPL",     "id":4784911, "ctx":"SignalHandler","msg":"Enqueuing the ReplicationStateTransitionLock for shutdown"}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"-",        "id":4784912, "ctx":"SignalHandler","msg":"Killing all operations for shutdown"}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"-",        "id":4695300, "ctx":"SignalHandler","msg":"Interrupted all currently running operations","attr":{"opsKilled":5}}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"TENANT_M", "id":5093807, "ctx":"SignalHandler","msg":"Shutting down all TenantMigrationAccessBlockers on global shutdown"}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"COMMAND",  "id":4784913, "ctx":"SignalHandler","msg":"Shutting down all open transactions"}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"REPL",     "id":4784914, "ctx":"SignalHandler","msg":"Acquiring the ReplicationStateTransitionLock for shutdown"}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"INDEX",    "id":4784915, "ctx":"SignalHandler","msg":"Shutting down the IndexBuildsCoordinator"}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"NETWORK",  "id":4784918, "ctx":"SignalHandler","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"REPL",     "id":4784920, "ctx":"SignalHandler","msg":"Shutting down the LogicalTimeValidator"}
{"t":{"$date":"2023-05-11T11:52:58.326+00:00"},"s":"I",  "c":"SHARDING", "id":4784921, "ctx":"SignalHandler","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"ASIO",     "id":22582,   "ctx":"MigrationUtil-TaskExecutor","msg":"Killing all outstanding egress activity."}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"COMMAND",  "id":4784923, "ctx":"SignalHandler","msg":"Shutting down the ServiceEntryPoint"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"CONTROL",  "id":4784925, "ctx":"SignalHandler","msg":"Shutting down free monitoring"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"CONTROL",  "id":20609,   "ctx":"SignalHandler","msg":"Shutting down free monitoring"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"CONTROL",  "id":4784927, "ctx":"SignalHandler","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"CONTROL",  "id":4784928, "ctx":"SignalHandler","msg":"Shutting down the TTL monitor"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"INDEX",    "id":3684100, "ctx":"SignalHandler","msg":"Shutting down TTL collection monitor thread"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"INDEX",    "id":3684101, "ctx":"SignalHandler","msg":"Finished shutting down TTL collection monitor thread"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"CONTROL",  "id":6278511, "ctx":"SignalHandler","msg":"Shutting down the Change Stream Expired Pre-images Remover"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"QUERY",    "id":6278515, "ctx":"SignalHandler","msg":"Shutting down Change Stream Expired Pre-images Remover thread"}
{"t":{"$date":"2023-05-11T11:52:58.327+00:00"},"s":"I",  "c":"QUERY",    "id":6278516, "ctx":"SignalHandler","msg":"Finished shutting down Change Stream Expired Pre-images Remover thread"}
{"t":{"$date":"2023-05-11T11:52:58.328+00:00"},"s":"I",  "c":"CONTROL",  "id":4784929, "ctx":"SignalHandler","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2023-05-11T11:52:58.328+00:00"},"s":"I",  "c":"CONTROL",  "id":4784930, "ctx":"SignalHandler","msg":"Shutting down the storage engine"}
{"t":{"$date":"2023-05-11T11:52:58.328+00:00"},"s":"I",  "c":"STORAGE",  "id":22320,   "ctx":"SignalHandler","msg":"Shutting down journal flusher thread"}
{"t":{"$date":"2023-05-11T11:52:58.328+00:00"},"s":"I",  "c":"STORAGE",  "id":22321,   "ctx":"SignalHandler","msg":"Finished shutting down journal flusher thread"}
{"t":{"$date":"2023-05-11T11:52:58.328+00:00"},"s":"I",  "c":"STORAGE",  "id":22322,   "ctx":"SignalHandler","msg":"Shutting down checkpoint thread"}
{"t":{"$date":"2023-05-11T11:52:58.328+00:00"},"s":"I",  "c":"STORAGE",  "id":22323,   "ctx":"SignalHandler","msg":"Finished shutting down checkpoint thread"}
{"t":{"$date":"2023-05-11T11:52:58.329+00:00"},"s":"I",  "c":"STORAGE",  "id":20282,   "ctx":"SignalHandler","msg":"Deregistering all the collections"}
{"t":{"$date":"2023-05-11T11:52:58.333+00:00"},"s":"I",  "c":"STORAGE",  "id":22372,   "ctx":"OplogVisibilityThread","msg":"Oplog visibility thread shutting down."}
{"t":{"$date":"2023-05-11T11:52:58.333+00:00"},"s":"I",  "c":"STORAGE",  "id":22261,   "ctx":"SignalHandler","msg":"Timestamp monitor shutting down"}
{"t":{"$date":"2023-05-11T11:52:58.334+00:00"},"s":"I",  "c":"STORAGE",  "id":22317,   "ctx":"SignalHandler","msg":"WiredTigerKVEngine shutting down"}
{"t":{"$date":"2023-05-11T11:52:58.334+00:00"},"s":"I",  "c":"STORAGE",  "id":22318,   "ctx":"SignalHandler","msg":"Shutting down session sweeper thread"}
{"t":{"$date":"2023-05-11T11:52:58.334+00:00"},"s":"I",  "c":"STORAGE",  "id":22319,   "ctx":"SignalHandler","msg":"Finished shutting down session sweeper thread"}
{"t":{"$date":"2023-05-11T11:52:58.335+00:00"},"s":"I",  "c":"STORAGE",  "id":4795902, "ctx":"SignalHandler","msg":"Closing WiredTiger","attr":{"closeConfig":"leak_memory=true,"}}
{"t":{"$date":"2023-05-11T11:52:58.443+00:00"},"s":"I",  "c":"STORAGE",  "id":4795901, "ctx":"SignalHandler","msg":"WiredTiger closed","attr":{"durationMillis":108}}
{"t":{"$date":"2023-05-11T11:52:58.443+00:00"},"s":"I",  "c":"STORAGE",  "id":22279,   "ctx":"SignalHandler","msg":"shutdown: removing fs lock..."}
{"t":{"$date":"2023-05-11T11:52:58.443+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"SignalHandler","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2023-05-11T11:52:58.443+00:00"},"s":"I",  "c":"FTDC",     "id":20626,   "ctx":"SignalHandler","msg":"Shutting down full-time diagnostic data capture"}
{"t":{"$date":"2023-05-11T11:52:58.447+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"SignalHandler","msg":"Now exiting"}
{"t":{"$date":"2023-05-11T11:52:58.448+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"SignalHandler","msg":"Shutting down","attr":{"exitCode":0}}

this seems to be the line of log show process termination which we don’t know why

Tracking down the source of the SIGTERM wiil be critical.

Could be a systemd issue. I would first isolate that by running sudo -u mongod mongod -f /etc/mongod.conf

And seeing as the kill comes from pid:1 I’m really leaning toward a systemd issue.

{"t":{"$date":"2023-05-11T11:52:27.806+00:00"},"s":"I",  "c":"CONTROL",  "id":23378,   "ctx":"SignalHandler","msg":"Signal was sent by kill(2)","attr":{"pid":1,"uid":0}}
1 Like

@chris

Thanks for the tips, no chance to verify that.
Do the same update today on the cluster and see that version 6.0.6 is available, use exactly the same procedure to update.
Once the “yum update mongo*” completed the mongod process is up (not even need to bring up using systemctl start mongd) and can see the standby is up normally.

I believe it’s 6.0.5 specific issues, we have now a 3 nodes RS running 6.0.6.

1 Like

Good to hear @Daniel_Wan1

Note to self, remember to check JIRA:
https://jira.mongodb.org/browse/SERVER-74345

Thanks Chris, JIRA is a place for fast checking. :+1: