Unable to restart or bring up one replica of MongoDB

Hi everyone,

This is the first time I am reaching out for help in this community, hope it goes well and my issue gets resolved asap. I really appreciate it if anyone can assist in getting these issues fixed.

I am using mongodb v5.0.18 in our application (Tyk gateway), as part of our tyk version upgrade we upgraded mongodb as well from 3.4.7 to 5.0.18 and since then there’s one or the other issue coming up.

Right now, there are two major issues…

  1. In our production env, we have mongodb deployed as a service in 3 servers (one primary and 2 replicas), I am able to bring up the primary and one replica but not the other replica, irrespective of I stop or kill or restart or do whatever, it is not coming up. I tried re-installing the entire mongodb but it is still the same.

below are the logs I got when I restarted…

{"t":{"$date":"2024-02-26T17:18:08.524+11:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"ReplWriterWorker-0","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1708928288:524267][19114:0x7fd280f16700], file:collection-36-8291676025468938374.wt, WT_SESSION.open_cursor: __posix_file_read, 435: /opt/app/mongo5/data/db/collection-36-8291676025468938374.wt: handle-read: pread: failed to read 282624 bytes at offset 6839042048: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-02-26T17:18:08.524+11:00"},"s":"F",  "c":"STORAGE",  "id":50882,   "ctx":"ReplWriterWorker-0","msg":"Failed to open WiredTiger cursor. This may be due to data corruption","attr":{"uri":"table:collection-36-8291676025468938374","config":"","error":{"code":8,"codeName":"UnknownError","errmsg":"-31802: WT_ERROR: non-specific WiredTiger error"},"message":"Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair"}}
{"t":{"$date":"2024-02-26T17:18:08.524+11:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"ReplWriterWorker-0","msg":"Fatal assertion","attr":{"msgid":50882,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp","line":109}}
{"t":{"$date":"2024-02-26T17:18:08.524+11:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"ReplWriterWorker-0","msg":"\n\n***aborting after fassert() failure\n\n"}
```

I restarted again with '--repair' on the dbpath as per the documentation and now it is failing with the below error..

```
{"t":{"$date":"2024-02-26T21:46:24.787+11:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":200}}
{"t":{"$date":"2024-02-26T21:46:24.789+11:00"},"s":"I",  "c":"REPL",     "id":21532,   "ctx":"initandlisten","msg":"Incremented the rollback ID","attr":{"rbid":28}}
{"t":{"$date":"2024-02-26T21:46:24.789+11:00"},"s":"F",  "c":"REPL",     "id":50923,   "ctx":"initandlisten","msg":"This instance has been repaired and may contain modified replicated data that would not match other replica set members. To see your repaired data, start mongod without the --replSet option. When you are finished recovering your data and would like to perform a complete re-sync, please refer to the documentation here: https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/"}
{"t":{"$date":"2024-02-26T21:46:24.789+11:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":50923,"file":"src/mongo/db/repl/replication_coordinator_impl.cpp","line":507}}
{"t":{"$date":"2024-02-26T21:46:24.789+11:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
```


2. the second issue is..
since the upgrade of mongodb to 5.0.18, our storage is continuously increasing and crossing the 50Gb storage limit and making the mongodb to fail. We've increased the storage to 150Gb and then also the same situation. I believe in the new 5.0.18, the log rotation is not happening properly which is leading to overload the storage. Below is my mongod.conf data..

```
net:
  bindIp: 0.0.0.0
  ipv6: false
  maxIncomingConnections: 51200
  port: 27017

processManagement:
  fork: true
  pidFilePath: /var/run/mongodb/mongod.pid

replication:
  oplogSizeMB: 1024
  replSetName: mongo-replica

security:
  authorization: disabled

storage:
  dbPath: /opt/app/mongo5/data/db
  engine: wiredTiger
  journal:
    enabled: true

systemLog:
  destination: file
  logAppend: true
  path: /opt/app/mongo5/log/mongod.log
  logRotate: reopen

i didn’t know you can do upgrading with such a big bump ?