Too many open files - Mongo 4.2

Niamh_Gibbons · September 1, 2020, 12:01pm

Hi Folks,

We have a MongoDB replica set configured - primary, secondary & arbiter. In the past few weeks one of the instances has crashed multiple times. The logs show the following :

2020-08-28T12:14:20.570+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.578+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.612+0000 I  SHARDING [conn54569] Marking collection *****.componenttypes as collection version: <unsharded>
2020-08-28T12:14:20.616+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.623+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.623+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.625+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.629+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.630+0000 W  NETWORK  [listener] Error accepting new connection TooManyFilesOpen: error in creating eventfd: Too many open files
2020-08-28T12:14:20.644+0000 I  NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2020-08-28T12:14:20.644+0000 I  NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2020-08-28T12:14:20.644+0000 I  NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2020-08-28T12:14:20.644+0000 I  NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files

The /lib/systemd/system/mongod.service file looks like this:

[Unit]
Description=An object/document-oriented database
Documentation=man:mongod(1)

[Service]
# Other directives omitted
# (file size)
LimitFSIZE=infinity
# (cpu time)
LimitCPU=infinity
# (virtual memory size)
LimitAS=infinity
# (open files)
LimitNOFILE=64000
# (processes/threads)
LimitNPROC=64000
User=mongodb
ExecStart=/usr/bin/mongod --config /etc/mongod.conf

[Install]
WantedBy=multi-user.target

The proc/pid/limits file is :

    Limit                     Soft Limit           Hard Limit           Units
    Max cpu time              unlimited            unlimited            seconds
    Max file size             unlimited            unlimited            bytes
    Max data size             unlimited            unlimited            bytes
    Max stack size            8388608              unlimited            bytes
    Max core file size        0                    unlimited            bytes
    Max resident set          unlimited            unlimited            bytes
    Max processes             64000                64000                processes
    Max open files            64000                64000                files
    Max locked memory         65536                65536                bytes
    Max address space         unlimited            unlimited            bytes
    Max file locks            unlimited            unlimited            locks
    Max pending signals       64122                64122                signals
    Max msgqueue size         819200               819200               bytes
    Max nice priority         0                    0
    Max realtime priority     0                    0
    Max realtime timeout      unlimited            unlimited            us

Any ideas what could be causing this?

Mongo version 4.2.8
Ubuntu 16.04

Thanks in advance.

Joe_Drumgoole · September 1, 2020, 12:24pm

You are hitting the operating system limit for the number of open file descriptors which is non unusual on database servers. Please refer to your operating system documentation for how to increase this limit.

Willy_Latorre · September 2, 2020, 5:21pm

you can check it, from unix doing an ulimit
$ ulimit -a ( and set the appropiate limit )

chris · September 2, 2020, 5:55pm

@Willy_Latorre Setting with the ulimit command only effects the shell session. This parameter need to be set in the appropriate script(or init system) that starts mongod.

ulimit is referenced in the Operations Checklist
Specifically in UNIX ulimit Settings

Niamh_Gibbons · September 16, 2020, 10:46am

Hi Guys,

Thanks for all the responses. I had already set the Mongo recommended ulimit settings in the service file and the proc limits file showed that the 64000 was set. However when I ran ulimit -u as the Mongo user, it was returning 1024. I increased this limit to 100000, but the instance has crashed again.

Steps to increase limit to 100000

Add fs.file-max = 100000 to /etc/sysctl.conf
Refresh with new config - sudo sysctl -p
Edit the /etc/security/limits.conf file & add following lines to it

'* soft nofile 100000
'* hard nofile 100000
mongodb soft nofile 100000
mongodb hard nofile 100000
root soft nofile 100000
root hard nofile 100000

Add ‘session required pam_limits.so’ to /etc/pam.d/common-session
Logout and login and ‘ulimit -a’ as Mongo user returns 100000

Any other ideas?

Cheers,
Niamh

chris · September 16, 2020, 11:37am

You missed the critical piece:

This means systemd, upstart or init script. There are examples toward the end of UNIX ulimit Settings for upstart and systemd.

ken.chen · September 16, 2020, 12:12pm

After modifying the service script, did you run systemctl daemon-reload?

Niamh_Gibbons · September 16, 2020, 1:23pm

Hey Chris,

Thanks for checking again, but I already have that in place in my /lib/systemd/system/mongod.service file…see original post.

I must be missing something else?

Thanks,
Niamh

chris · September 16, 2020, 1:25pm

Yep. See @ken.chen comment.

This might need to be updated in the documentation too.

Niamh_Gibbons · September 16, 2020, 5:38pm

Hey Ken / Chris,

The config in the service file has been in place for a long time, I inherited the system like this Although I recently upgraded this Mongo instance and issued systemctl daemon-reload at the Disable Transparent Huge Pages (THP) step, where I created the /etc/systemd/system/disable-transparent-huge-pages.service file.

Any other suggestions as to what I could be missing? I’m new to Mongo and your help is really appreciated.

Thanks,
Niamh

Niamh_Gibbons · September 30, 2020, 4:24pm

Hi Folks

This mongo instance has crashed a few more times since, with the below in the logs the latest time. Any other troubleshooting advice?

**2020-09-30T11:05:40.172+0000 I  ACCESS   [conn94647] Successfully authenticated as principal DbUser on admin from client 13.0.1.189:34528
2020-09-30T11:05:40.172+0000 I  ACCESS   [conn94559] Successfully authenticated as principal DbUser on admin from client 13.0.1.189:34356
2020-09-30T11:05:40.180+0000 I  ACCESS   [conn94566] Successfully authenticated as principal DbUser on admin from client 13.0.1.189:34358
2020-09-30T11:05:40.181+0000 E  -        [conn95032] cannot open /dev/urandom Too many open files
2020-09-30T11:05:40.181+0000 F  -        [conn95032] Fatal Assertion 28839 at src/mongo/platform/random.cpp 159
2020-09-30T11:05:40.183+0000 F  -        [conn95032]

***aborting after fassert() failure


2020-09-30T11:05:40.183+0000 E  -        [conn95028] cannot open /dev/urandom Too many open files
2020-09-30T11:05:40.183+0000 F  -        [conn95028] Fatal Assertion 28839 at src/mongo/platform/random.cpp 159
2020-09-30T11:05:40.183+0000 F  -        [conn95028]

***aborting after fassert() failure


2020-09-30T11:05:40.183+0000 E  -        [conn95031] cannot open /dev/urandom Too many open files
2020-09-30T11:05:40.183+0000 F  -        [conn95031] Fatal Assertion 28839 at src/mongo/platform/random.cpp 159
2020-09-30T11:05:40.183+0000 F  -        [conn95031]

***aborting after fassert() failure


2020-09-30T11:05:40.183+0000 E  -        [conn95025] cannot open /dev/urandom Too many open files
2020-09-30T11:05:40.183+0000 F  -        [conn95025] Fatal Assertion 28839 at src/mongo/platform/random.cpp 159
2020-09-30T11:05:40.183+0000 F  -        [conn95025]

***aborting after fassert() failure**

Cheers,
Niamh

chris · September 30, 2020, 4:32pm

Increase nofile limit further.

You should inspect how many connections you have open too.

Poorly written clients will consume open files due to an open socket == file handle.

Niamh_Gibbons · October 5, 2020, 8:34am

Thanks Chris. There seems to be thousands of open connections coming from the app. I looked into it further and can see the mongoose driver is outdated so getting that updated and hope it helps. Thanks for the help with this

Niamh_Gibbons · November 9, 2020, 11:35am

Hi folks - updating the mongoose driver seems to have done the trick! Thanks all for the help/advice

system · November 14, 2020, 11:35am

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.