Mongo no longer accepting the connection

We have a standalone MongoDB server and for some reason we are seeing that Mongodb is no longer accepting any new connections the weird part is that

I’m attempting to make a Mongodb connection

mongo ‘mongodb://’

Received Timeout

connecting to: mongodb://
Error: couldn't connect to server, connection attempt failed: SocketException: Error connecting to :: caused by :: Connection timed out :
exception: connect failed
exiting with code 1

I see the following result

but nothing is visible on the mongodb.log

The only thing is visible are …

iredTiger message","attr":{"message":"[1688542311:637059][10933:0x7f3a7403f700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 341775, snapshot max: 341827 snapshot count: 3, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0)"}}
{"t":{"$date":"2023-07-05T07:32:02.744+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"","connectionId":19534,"connectionCount":77}}
{"t":{"$date":"2023-07-05T07:40:00.570+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"","connectionId":19535,"connectionCount":78}}
{"t":{"$date":"2023-07-05T08:40:17.003+00:00"},"s":"I",  "c":"-",        "id":20883,   "ctx":"conn20","msg":"Interrupted operation as its client disconnected","attr":{"opId":1854086}}
{"t":{"$date":"2023-07-05T08:40:17.009+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn20","msg":"Connection ended","attr":{"remote":"","connectionId":20,"connectionCount":78}}

I can see that Connection Count is 78 but I don’t know the max value for it.

I suspect this could be because of Max Open Files but I doubt it because of the following
a) No such error is visible on the logs
b) The clients are timing out instead of returning failure immediately.
c) the systemd unit has the value set to 64000

I have a feeling if I restart the server everything will become normal but we would like to know
we can prevent such problem in future.

And I was right after the restart things did work :slight_smile:

Also, I’m not sure what the max open files limit for Mongodb user from systemclt unit it seems like 64000

# file size
# cpu time
# virtual memory size
# open files
# processes/threads
# locked memory
# total threads (user+kernel)
# mongod.conf

  dbPath: /var/lib/mongodb
    enabled: true

  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

  port: 27017

  timeZoneInfo: /usr/share/zoneinfo

Mongo db version we have

db version v4.4.5
Build Info: {
    "version": "4.4.5",
    "gitVersion": "ff5cb77101b052fa02da43b8538093486cf9b3f7",
    "openSSLVersion": "OpenSSL 1.1.1n  15 Mar 2022",
    "modules": [],
    "allocator": "tcmalloc",
    "environment": {
        "distmod": "debian10",
        "distarch": "x86_64",
        "target_arch": "x86_64"

Mongo shell version

MongoDB shell version v4.4.5
Build Info: {
    "version": "4.4.5",
    "gitVersion": "ff5cb77101b052fa02da43b8538093486cf9b3f7",
    "openSSLVersion": "OpenSSL 1.1.1n  15 Mar 2022",
    "modules": [],
    "allocator": "tcmalloc",
    "environment": {
        "distmod": "debian10",
        "distarch": "x86_64",
        "target_arch": "x86_64"

I can confirm with systemd unit file the File limit is set to 640000

Did you see anything useful in driver logs? it might be in the driver side if the connection request never reaches the server side. e.g. the driver has run out of connections in the pool and a new request has to wait for one. (and eventually times out)

@Kobe_W : problem persisted on every service that was trying to establish connection with mongoDB

  • Nodejs
  • golang
  • mongo shell

All services or commands were timing out with no explanation on mongo logs.

Did you try to set the logLevel to something higher? The higher the number (1-5) the more logs you see. This might show a hidden message or something else going on in the future. Just note it generates a lot of logs the higher you go so don’t keep it on all the time. Only if you are seeing a problem.

@tapiocaPENGUIN I believe even for that mongo shell has to connect which was not happening at all.

As this was a production server I could not have let this happen for too long as it was affecting our other services and I had to restart the MongoDB server which solved the issue.

My intention was to understand if there is anyone that has seen issue something similar which could help us in future if we are affected with the problem again.