Pthread_create failed: Resource temporarily unavailable in mongo

Ops and Admin
1

Hi Guys,

We are running 32k connections on mongodb so facing below error on log file. After suggestions from mongo community to check pid_max and threads-max having little bit high only and number of connections are opened which means sockets are quite high and not closed like below.

  1. Centos 8.1 (4.18.0-193.14.2.el8_2.x86_64)
  2. Mongo version 3.6.17

cat /proc/sys/kernel/pid_max 4194304
cat /proc/sys/kernel/threads-max 94465

mongod 8955 root *366u IPv4 120213606 0t0 TCP testmanager:33445->node03:49816 (CLOSE_WAIT)
mongod 8955 root *367u IPv4 120213789 0t0 TCP testmanager:33445->node03:49860 (CLOSE_WAIT)
mongod 8955 root *368u IPv4 120402126 0t0 TCP testmanager:33445->node03:49864 (CLOSE_WAIT)
mongod 8955 root *369u IPv4 120437763 0t0 TCP testmanager:33445->node03:49866 (CLOSE_WAIT)

After some time socket descriptors reaching max limit. And mongo throwing an errror for thread creation.

2021-02-24T22:50:04.692+0000 I -        [listener] pthread_create failed: Resource temporarily unavailable
2021-02-24T22:50:04.692+0000 W EXECUTOR [conn480782] Terminating session due to error: InternalError: failed to create service entry worker thread
2021-02-24T22:50:05.589+0000 I -        [listener] pthread_create failed: Resource temporarily unavailable
2021-02-24T22:50:05.589+0000 W EXECUTOR [conn480783] Terminating session due to error: InternalError: failed to create service entry worker thread

https://jira.mongodb.org/browse/SERVER-17687

Below observation copied from above jira ticket.

If the issue is not the system-wide limit on the number of threads then the resource exhaustion is somewhere else. You’ll need to investigate what resource is being exhausted (memory and number of file descriptors / sockets are the usual suspects) or simply lower the number of threads. If you’re not using connection pooling you’re probably running out of sockets (netstat -a | grep TIME_WAIT may help).

As per analysis, sockets descriptors getting exhausted and mongo thread creating is getting failed. Any suggestions why sockets are not getting closed or any workaround for this.

2

I don’t know if this is the same issue, but it may help.

Our Mongo instances started to crash after an upgrade from Ubuntu 16.04 to 18.04. The only error in the > Mongo logs was the same as above:

2021-04-15T10:53:15.304+0000 I  -        [listener] pthread_create failed: Resource temporarily unavailable
2021-04-15T10:53:15.305+0000 W  EXECUTOR [conn12574] Terminating session due to error: InternalError: failed to create service entry worker thread

I also found the following log in syslog:

Apr 15 10:53:15 mongo-server kernel: [65474.378840] cgroup: fork rejected by pids controller in /system.slice/mongod.service

So I checked the service again and could see the tasks limit was set to 4618:

my-user@mongo-server:~$ sudo service mongod status
● mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/lib/systemd/system/mongod.service; disabled; vendor preset: enabled)
Active: active (running) since Wed 2021-02-24 15:21:38 UTC; 1 months 20 days ago
Main PID: 17990 (mongod)
Tasks: 287 (limit: 4618)
CGroup: /system.slice/mongod.service
└─17990 /usr/bin/mongod --quiet --config /etc/mongod.conf

I changed this in the /lib/systemd/system/mongod.service file to unlimited by adding the below within [Service]

total threads (user+kernel)

TasksMax=infinity

Then ran the below on all Mongo instances and this removed the Task limit

sudo systemctl daemon-reload
sudo service mongod restart

This has worked for us so far, hopefully it helps someone else.

1 Like