I have a website with about 200 requests per second. I configured a replica set with 3 nodes of mongod 4.2.15. The processes are running inside supervisord:
[program:mongo_27017]
command=/opt/mongodb-linux-x86_64-debian10-4.2.15/bin/mongod --port 27017 --dbpath=/storage/mongo_data_27017 --bind_ip 127.0.0.1,192.168.0.16 --replSet rs0 --keyFile=/home/admin/keys/mongo-rs0-keyfile
directory=/home/admin
user=admin
autostart=true
I still cannot figure out why this happened. Also with the same configuration, 3 months ago I used version 3.6 without any problems. That time my website had even more requests than today.
Currently I am really stuck on this. Thank you for any help you can provide.
# log in secondary when restart
2021-09-17T18:00:03.513+0000 I NETWORK [listener] Error accepting new connection on 192.168.0.11:27017: Too many open files
2021-09-17T18:00:03.513+0000 I NETWORK [listener] Error accepting new connection on 192.168.0.11:27017: Too many open files
2021-09-17T18:00:03.513+0000 I NETWORK [listener] Error accepting new connection on 192.168.0.11:27017: Too many open files
# open file limit on secondary node
ulimit -Sn
500000
ulimit -Hn
500000
Primary at restarted time:
2021-09-17T18:07:38.961+0000 I COMMAND [ftdc] serverStatus was very slow: { after basic: 1, after asserts: 1, after connections: 1, after electionMetrics: 24, after extra_info: 78, after flowControl: 78, after globalLock: 78, after locks: 78, after logicalSessionRecordCache: 90, after network: 90, after opLatencies: 90, after opReadConcernCounters: 90, after opcounters: 90, after opcountersRepl: 90, after oplogTruncation: 126, after repl: 144, after security: 144, after storageEngine: 157, after tcmalloc: 341, after trafficRecording: 391, after transactions: 391, after transportSecurity: 391, after twoPhaseCommitCoordinator: 449, after wiredTiger: 921, at end: 2885 }
2021-09-17T18:07:47.220+0000 I CONTROL [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2021-09-17T18:07:47.224+0000 W ASIO [main] No TransportLayer configured during NetworkInterface startup
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] MongoDB starting : pid=8993 port=27020 dbpath=/storage/mongo_data_27020 64-bit host=nganvu
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] db version v4.2.15
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] git version: d7fd78dead621a539c20791a93abec34bb1be385
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] allocator: tcmalloc
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] modules: none
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] build environment:
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] distmod: debian10
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] distarch: x86_64
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] target_arch: x86_64
2021-09-17T18:07:47.309+0000 I CONTROL [initandlisten] options: { net: { bindIp: "127.0.0.1,192.168.0.9", port: 27020 }, replication: { replSet: "rs0" }, security: { keyFile: "/home/admin/keys/mongo-rs0-keyfile" }, storage: { dbPath: "/storage/mongo_data" }, systemLog: { quiet: true } }
2021-09-17T18:07:47.310+0000 W STORAGE [initandlisten] Detected unclean shutdown - /storage/mongo_data/mongod.lock is not empty.