MongoDB does not open Port

Hello,

We are running a MongoDB server at Debian 10.13.
The MongoDB version is 4.4.19.
After the restart of one MongoDB server, the process does not open a network port anymore.
I have already checked the configuration file:

net:
  port: 27017
  bindIp: localhost,127.0.0.1

Even when I try to use another port it does not work.
The mondogb user and group is the owner of the dataarea and of the socket.
I already tried to use the repair command but after it starts, it does nothing to the database, even after running for up to 12 hours.

When I unmap the dataarea, a new one will be created and the server starts as I would expect it.
I tried to use a backup of the server (without restoring the DB) to check whether there has been a change to the os which prevents the server from opening the port.

In the log files you can see this:

{"t":{"$date":"2023-03-02T09:16:03.095Z"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2023-03-02T09:16:03.112Z"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2023-03-02T09:16:03.112Z"},"s":"D2", "c":"CONNPOOL", "id":22558,   "ctx":"main","msg":"Initializing connection pool controller","attr":{"pool":"NetworkInterfaceTL-ReplNetwork","controller":"LimitController"}}
{"t":{"$date":"2023-03-02T09:16:03.112Z"},"s":"D1", "c":"NETWORK",  "id":22940,   "ctx":"main","msg":"file descriptor and connection resource limits","attr":{"hard":64000,"soft":64000,"conn":51200}}
{"t":{"$date":"2023-03-02T09:16:03.112Z"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":31002,"port":27017,"dbPath":"/var/lib/mongodb","architecture":"64-bit","host":"linsrv150"}}
{"t":{"$date":"2023-03-02T09:16:03.112Z"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.19","gitVersion":"9a996e0ad993148b9650dc402e6d3b1804ad3b8a","openSSLVersion":"OpenSSL 1.1.1n  15 Mar 2022","modules":[],"allocator":"tcmalloc","environment":{"distmod":"debian10","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2023-03-02T09:16:03.113Z"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","version":"Kernel 4.19.0-23-amd64"}}}
{"t":{"$date":"2023-03-02T09:16:03.113Z"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"localhost,127.0.0.1","port":27017},"processManagement":{"timeZoneInfo":"/usr/share/zoneinfo"},"storage":{"dbPath":"/var/lib/mongodb","directoryPerDB":true,"engine":"wiredTiger","journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":false,"path":"/var/log/mongodb/mongod.log","timeStampFormat":"iso8601-utc","verbosity":5}}}}
{"t":{"$date":"2023-03-02T09:16:03.113Z"},"s":"D1", "c":"NETWORK",  "id":22940,   "ctx":"initandlisten","msg":"file descriptor and connection resource limits","attr":{"hard":64000,"soft":64000,"conn":51200}}
{"t":{"$date":"2023-03-02T09:16:03.113Z"},"s":"D1", "c":"EXECUTOR", "id":23104,   "ctx":"OCSPManagerHTTP-0","msg":"Starting thread","attr":{"threadName":"OCSPManagerHTTP-0","poolName":"OCSPManagerHTTP"}}
{"t":{"$date":"2023-03-02T09:16:03.113Z"},"s":"D3", "c":"EXECUTOR", "id":23108,   "ctx":"OCSPManagerHTTP-0","msg":"Waiting for work","attr":{"numThreads":1,"minThreads":1}}
{"t":{"$date":"2023-03-02T09:16:03.113Z"},"s":"D2", "c":"-",        "id":23323,   "ctx":"initandlisten","msg":"Starting periodic job {job_name}","attr":{"job_name":"FlowControlRefresher"}}
{"t":{"$date":"2023-03-02T09:16:03.113Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-02T09:16:03.114Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}
{"t":{"$date":"2023-03-02T09:16:04.113Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-02T09:16:04.113Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}
{"t":{"$date":"2023-03-02T09:16:05.113Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-02T09:16:05.113Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}
{"t":{"$date":"2023-03-02T09:16:06.113Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-02T09:16:06.113Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}
{"t":{"$date":"2023-03-02T09:16:07.113Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-02T09:16:07.113Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}

Even reinstalling the mongodb packages does not solve the issue.
Do you have any idea?

Thanks for your help!
Best regards
Florian

Hello @Florian_Streppel ,

Welcome to The MongoDB Community Forums! :wave:

To understand your use-case better, please share more details such as:

  • How did you determine that the port was not opened? Was there an error in the logs?
  • What is your deployment topology? If it is a replica set please share the output of rs.status() and rs.conf()
  • By dataarea did you mean the dbpath? What does unmap means in this context?
  • I don’t see any errors in the logs you shared, could you please share the error you are getting?

Warning: Only use mongod --repair if you have no other options. The operation removes and does not save any corrupt data during the repair process and should not be run in a replica set as mentioned in the repairDatabase manual.

If you’re having issues starting and connecting to a self-hosted MongoDB deployment, you might find hints in the following topic:

Regards,
Tarun

Hello Tarun,

sorry for my late reply.

  • How did you determine that the port was not opened? Was there an error in the logs?

When I let me show the open ports on the server there is no mongodb port opened.
In the logfiles it look like the port is opened. So there are no hints in the log for me.

  • What is your deployment topology? If it is a replica set please share the output of rs.status() and rs.conf()

I cannot share the output because I cannot even connect to the server with the cli.

  • By dataarea did you mean the dbpath? What does unmap means in this context?

Yes I mean the dbpath. The dbpath is a mapped nfs share and I unmounted the share to test, whether there is an issue with the data.

  • I don’t see any errors in the logs you shared, could you please share the error you are getting?

Thats my issue as well. I cannot see any error. But the mongod process does not open a port I can connect to.

Best regards
Florian

Did you run something like netstat ?

Yes I ran netstat -tulpen but there is no port opened for mongod.

Please post the output of the below to investigate further:

systemctl status mongod.service

telnet 127.0.0.1 27017

netstat -an | grep LISTEN

systemctl status ufw.service

iptables -L

systemctl status mongod
● mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2023-03-22 09:29:51 CET; 13s ago
     Docs: https://docs.mongodb.org/manual
 Main PID: 20523 (mongod)
   Memory: 16.1M
   CGroup: /system.slice/mongod.service
           └─20523 /usr/bin/mongod --config /etc/mongod.conf

Mär 22 09:29:51 linsrv150 systemd[1]: Started MongoDB Database Server.
Mär 22 09:29:51 linsrv150 mongod[20523]: {"t":{"$date":"2023-03-22T08:29:51.761Z"},"s":"I",  "c":"CONTROL",  "id":20697,   "ctx":"main","msg":"Renamed existing log file","attr":{"oldLogPath":"/var/log/mongodb/mon


telnet 127.0.0.1 27017
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

netstat -an |grep LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:60623           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:42131           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp6       0      0 :::46575                :::*                    LISTEN
tcp6       0      0 :::111                  :::*                    LISTEN
tcp6       0      0 :::56625                :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN


systemctl status ufw
Unit ufw.service could not be found.

iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Please share the content of

and of the log file. I cannot tell you the exact path of the log file because for some reason the message is truncated.

1 dumb question: Are you trying to connect from the same machine where you start mongodb or from another machine?

storage:
  dbPath: /var/lib/mongodb
  directoryPerDB: true
  engine: wiredTiger
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: false
  verbosity: 5
  path: /var/log/mongodb/mongod.log
  timeStampFormat: iso8601-utc

# network interfaces
net:
  port: 27017
  bindIp: localhost,127.0.0.1,<private IP>

# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

{"t":{"$date":"2023-03-22T08:29:51.763Z"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"D2", "c":"CONNPOOL", "id":22558,   "ctx":"main","msg":"Initializing connection pool controller","attr":{"pool":"NetworkInterfaceTL-ReplNetwork","controller":"LimitController"}}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"D1", "c":"NETWORK",  "id":22940,   "ctx":"main","msg":"file descriptor and connection resource limits","attr":{"hard":64000,"soft":64000,"conn":51200}}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":20523,"port":27017,"dbPath":"/var/lib/mongodb","architecture":"64-bit","host":"linsrv150"}}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.19","gitVersion":"9a996e0ad993148b9650dc402e6d3b1804ad3b8a","openSSLVersion":"OpenSSL 1.1.1n  15 Mar 2022","modules":[],"allocator":"tcmalloc","environment":{"distmod":"debian10","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","version":"Kernel 4.19.0-23-amd64"}}}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"localhost,127.0.0.1,10.54.9.10","port":27017},"processManagement":{"timeZoneInfo":"/usr/share/zoneinfo"},"storage":{"dbPath":"/var/lib/mongodb","directoryPerDB":true,"engine":"wiredTiger","journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":false,"path":"/var/log/mongodb/mongod.log","timeStampFormat":"iso8601-utc","verbosity":5}}}}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"D1", "c":"NETWORK",  "id":22940,   "ctx":"initandlisten","msg":"file descriptor and connection resource limits","attr":{"hard":64000,"soft":64000,"conn":51200}}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"D1", "c":"EXECUTOR", "id":23104,   "ctx":"OCSPManagerHTTP-0","msg":"Starting thread","attr":{"threadName":"OCSPManagerHTTP-0","poolName":"OCSPManagerHTTP"}}
{"t":{"$date":"2023-03-22T08:29:51.836Z"},"s":"D3", "c":"EXECUTOR", "id":23108,   "ctx":"OCSPManagerHTTP-0","msg":"Waiting for work","attr":{"numThreads":1,"minThreads":1}}
{"t":{"$date":"2023-03-22T08:29:51.838Z"},"s":"D2", "c":"-",        "id":23323,   "ctx":"initandlisten","msg":"Starting periodic job {job_name}","attr":{"job_name":"FlowControlRefresher"}}
{"t":{"$date":"2023-03-22T08:29:51.838Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-22T08:29:51.838Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}
{"t":{"$date":"2023-03-22T08:29:52.838Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-22T08:29:52.838Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}
{"t":{"$date":"2023-03-22T08:29:53.838Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-22T08:29:53.838Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}
{"t":{"$date":"2023-03-22T08:29:54.838Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-22T08:29:54.838Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}
{"t":{"$date":"2023-03-22T08:29:55.838Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}
{"t":{"$date":"2023-03-22T08:29:55.838Z"},"s":"D4", "c":"-",        "id":20518,   "ctx":"FlowControlRefresher","msg":"Refreshing tickets. Before: {tickets} Now: {numTickets}","attr":{"tickets":1000000000,"numTickets":1000000000}}
{"t":{"$date":"2023-03-22T08:29:56.838Z"},"s":"D4", "c":"STORAGE",  "id":22222,   "ctx":"FlowControlRefresher","msg":"Trimmed samples. Num: {numTrimmed}","attr":{"numTrimmed":0}}

I try to connect from the same machine.
I just connect for monitoring from remote.

@Florian_Streppel

For troubleshooting, please set the bindIP as:

bindIp: 0.0.0.0

Restart the mongod services and try to connect first locally and then remotely

1 Like

I did change it:

netstat -an |grep LISTEN
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:60623           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:42131           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp6       0      0 :::46575                :::*                    LISTEN
tcp6       0      0 :::111                  :::*                    LISTEN
tcp6       0      0 :::56625                :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN

But there ist still no open port.
When I try to connect with telnet it does not work as well.

Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused

@Florian_Streppel Please let me know by which user the mongod service is running

ps -ef | grep mongod

It is running with the mongodb user:

mongodb  18573     1  0 15:04 ?        00:00:00 /usr/bin/mongod --config /etc/mongod.conf

The dbpath is owned by the mongodb user.

When you redact

redact all

What else did you redact? May be it stops us from investigating.

Please share output of

ls -l /var/lib/mongodb/*

If your data set is huge, I mean huge, you might need to let mongod open all the files before it starts to listen.

Are you running SELinux?

Is your /var/lib/mongodb storage NAS, SAN or local?

In addition to what Steeve has asked, do you have free space in dbPath directory to write Journal files?

Did the directory get mounted as read write after reboot? Because as you have mentioned earlier, the problem started after you rebooted the machine.

Check if there are any mount points as read only (ro)

cat /proc/mounts

As Steeve mentioned, check for SELinux if enabled

getenforce

1 Like
"ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","version":"Kernel 4.19.0-23-amd64"}}}

it is Debian, so not a Mac issue.

I believe you were quick to judge, are looking at the wrong place and thus leading us to the wrong ideas. MongoDB will start serving at that port only when everything is fine, else will shutdown with an error. And, as happens all the time, there can be certain cases that are not anticipated by developers that cause lingering states without a timeout. And I believe this is one of those situations: Your config file is fine so the server passes the initial phases, but gets into a lingering state at storage stage "s":"D4", "c":"STORAGE"

Here, you should now have some new sights. The problem is in the NFS portion of your setup and in one of these sections: nfs driver on this pc (did you changed any related settings?), network itself (any heavy load recently added that slow network?), nfs data host pc (is nfs host healthy?), nfs shared data drive/folder (is data hard drive healthy? data folder?). check all these parts and those I could not think of that are in this pipeline.

PS: as noted above, when you attach/write log files, check for possible sensitive information in them too.

2 Likes

Hi there,

I had conact with the MongoDB Support and we solved the issue.
We are running MongoDB with the DBPath set to an NFS share.
The server is able to access the share and can create and read files there.

When I unmounted the NFS share and started MongoDB again, it was working.
The DBPath was now pointing on the partition on the server itself.
After this I created an empty NFS Share and tried it there as well, but with the result, that it did not work.
No database was created. Just the mongod.lock file was written.

I found this error message in /var/log/syslog.

Apr 24 10:44:24 Server kernel: [4830863.552178] lockd: server <NFS Server> not responding, still trying

I added a rule at the firewall between the server and the NFS server that allowed anything.
After restarting the service it did work and I got this message in the syslog.

Apr 24 10:44:24 Server kernel: [4830863.554797] lockd: server <NFS Server> OK

In the log of the Firewall I found a connection from the MongoDB server to the NFS server on port 4045 UDP.
The Port is used for NFS lock daemon/manager.
I added the port to the firewall rule and restarted the service.

Now everything is working just fine again.

Thank you for having a look at our issue.

Best regards Florian

3 Likes