Hi everyone,
This is the first time I am reaching out for help in this community, hope it goes well and my issue gets resolved asap. I really appreciate it if anyone can assist in getting these issues fixed.
I am using mongodb v5.0.18 in our application (Tyk gateway), as part of our tyk version upgrade we upgraded mongodb as well from 3.4.7 to 5.0.18 and since then there’s one or the other issue coming up.
Right now, there are two major issues…
- In our production env, we have mongodb deployed as a service in 3 servers (one primary and 2 replicas), I am able to bring up the primary and one replica but not the other replica, irrespective of I stop or kill or restart or do whatever, it is not coming up. I tried re-installing the entire mongodb but it is still the same.
below are the logs I got when I restarted…
{"t":{"$date":"2024-02-26T17:18:08.524+11:00"},"s":"E", "c":"STORAGE", "id":22435, "ctx":"ReplWriterWorker-0","msg":"WiredTiger error","attr":{"error":-31802,"message":"[1708928288:524267][19114:0x7fd280f16700], file:collection-36-8291676025468938374.wt, WT_SESSION.open_cursor: __posix_file_read, 435: /opt/app/mongo5/data/db/collection-36-8291676025468938374.wt: handle-read: pread: failed to read 282624 bytes at offset 6839042048: WT_ERROR: non-specific WiredTiger error"}}
{"t":{"$date":"2024-02-26T17:18:08.524+11:00"},"s":"F", "c":"STORAGE", "id":50882, "ctx":"ReplWriterWorker-0","msg":"Failed to open WiredTiger cursor. This may be due to data corruption","attr":{"uri":"table:collection-36-8291676025468938374","config":"","error":{"code":8,"codeName":"UnknownError","errmsg":"-31802: WT_ERROR: non-specific WiredTiger error"},"message":"Please read the documentation for starting MongoDB with --repair here: http://dochub.mongodb.org/core/repair"}}
{"t":{"$date":"2024-02-26T17:18:08.524+11:00"},"s":"F", "c":"-", "id":23091, "ctx":"ReplWriterWorker-0","msg":"Fatal assertion","attr":{"msgid":50882,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_session_cache.cpp","line":109}}
{"t":{"$date":"2024-02-26T17:18:08.524+11:00"},"s":"F", "c":"-", "id":23092, "ctx":"ReplWriterWorker-0","msg":"\n\n***aborting after fassert() failure\n\n"}
```
I restarted again with '--repair' on the dbpath as per the documentation and now it is failing with the below error..
```
{"t":{"$date":"2024-02-26T21:46:24.787+11:00"},"s":"I", "c":"-", "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"ReadConcernMajorityNotAvailableYet: Read concern majority reads are currently not possible.","nextWakeupMillis":200}}
{"t":{"$date":"2024-02-26T21:46:24.789+11:00"},"s":"I", "c":"REPL", "id":21532, "ctx":"initandlisten","msg":"Incremented the rollback ID","attr":{"rbid":28}}
{"t":{"$date":"2024-02-26T21:46:24.789+11:00"},"s":"F", "c":"REPL", "id":50923, "ctx":"initandlisten","msg":"This instance has been repaired and may contain modified replicated data that would not match other replica set members. To see your repaired data, start mongod without the --replSet option. When you are finished recovering your data and would like to perform a complete re-sync, please refer to the documentation here: https://docs.mongodb.com/manual/tutorial/resync-replica-set-member/"}
{"t":{"$date":"2024-02-26T21:46:24.789+11:00"},"s":"F", "c":"-", "id":23091, "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":50923,"file":"src/mongo/db/repl/replication_coordinator_impl.cpp","line":507}}
{"t":{"$date":"2024-02-26T21:46:24.789+11:00"},"s":"F", "c":"-", "id":23092, "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}
```
2. the second issue is..
since the upgrade of mongodb to 5.0.18, our storage is continuously increasing and crossing the 50Gb storage limit and making the mongodb to fail. We've increased the storage to 150Gb and then also the same situation. I believe in the new 5.0.18, the log rotation is not happening properly which is leading to overload the storage. Below is my mongod.conf data..
```
net:
bindIp: 0.0.0.0
ipv6: false
maxIncomingConnections: 51200
port: 27017
processManagement:
fork: true
pidFilePath: /var/run/mongodb/mongod.pid
replication:
oplogSizeMB: 1024
replSetName: mongo-replica
security:
authorization: disabled
storage:
dbPath: /opt/app/mongo5/data/db
engine: wiredTiger
journal:
enabled: true
systemLog:
destination: file
logAppend: true
path: /opt/app/mongo5/log/mongod.log
logRotate: reopen