MongoDB restart several times a day

I have a website with about 200 requests per second. I configured a replica set with 3 nodes of mongod 4.2.15. The processes are running inside supervisord:

command=/opt/mongodb-linux-x86_64-debian10-4.2.15/bin/mongod --port 27017 --dbpath=/storage/mongo_data_27017 --bind_ip, --replSet rs0 --keyFile=/home/admin/keys/mongo-rs0-keyfile

I still cannot figure out why this happened. Also with the same configuration, 3 months ago I used version 3.6 without any problems. That time my website had even more requests than today.

Currently I am really stuck on this. Thank you for any help you can provide.

# log in secondary when restart
2021-09-17T18:00:03.513+0000 I  NETWORK  [listener] Error accepting new connection on Too many open files
2021-09-17T18:00:03.513+0000 I  NETWORK  [listener] Error accepting new connection on Too many open files
2021-09-17T18:00:03.513+0000 I  NETWORK  [listener] Error accepting new connection on Too many open files

# open file limit on secondary node
ulimit -Sn
ulimit -Hn

Primary at restarted time:

2021-09-17T18:07:38.961+0000 I  COMMAND  [ftdc] serverStatus was very slow: { after basic: 1, after asserts: 1, after connections: 1, after electionMetrics: 24, after extra_info: 78, after flowControl: 78, after globalLock: 78, after locks: 78, after logicalSessionRecordCache: 90, after network: 90, after opLatencies: 90, after opReadConcernCounters: 90, after opcounters: 90, after opcountersRepl: 90, after oplogTruncation: 126, after repl: 144, after security: 144, after storageEngine: 157, after tcmalloc: 341, after trafficRecording: 391, after transactions: 391, after transportSecurity: 391, after twoPhaseCommitCoordinator: 449, after wiredTiger: 921, at end: 2885 }
2021-09-17T18:07:47.220+0000 I  CONTROL  [main] Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'
2021-09-17T18:07:47.224+0000 W  ASIO     [main] No TransportLayer configured during NetworkInterface startup
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten] MongoDB starting : pid=8993 port=27020 dbpath=/storage/mongo_data_27020 64-bit host=nganvu
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten] db version v4.2.15
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten] git version: d7fd78dead621a539c20791a93abec34bb1be385
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten] allocator: tcmalloc
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten] modules: none
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten] build environment:
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten]     distmod: debian10
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten]     distarch: x86_64
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten]     target_arch: x86_64
2021-09-17T18:07:47.309+0000 I  CONTROL  [initandlisten] options: { net: { bindIp: ",", port: 27020 }, replication: { replSet: "rs0" }, security: { keyFile: "/home/admin/keys/mongo-rs0-keyfile" }, storage: { dbPath: "/storage/mongo_data" }, systemLog: { quiet: true } }
2021-09-17T18:07:47.310+0000 W  STORAGE  [initandlisten] Detected unclean shutdown - /storage/mongo_data/mongod.lock is not empty.

Doing this in the shell and expecting it to apply elsewhere is a common misconception, it only applies to the current shell. The ulimit values needs to be set when mongod is invoked. Usually this is done in the systemd target or the SystemV init script. You will have to configure it for supervisord, I’m of no help here :slight_smile:

The current limits for your running mongod are in /proc/$(pgrep mongod)/limits

Max Open files applies to all the file handler used by mongodb: Open data files, index files, journal files and open sockets(connections).


Yes I got it. Currently I have switched to systemd with default installation from documents of MongoDB. Run cat the process mongod increases the open files limit to 64000 by default.

I need several days to confirm the stability of this configuration.

Thank you.

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.