Mongodb on Ubuntu -' Main process exited, code=exited, status=1/FAILURE'

Konstantina_Arnaoutaki · March 22, 2022, 2:34pm

Mongo has suddenly stopped working, so I am trying to restart it I’ am getting again and again though errors like the below:

 mongod.service - MongoDB Database Server
   Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Τρι 2022-03-22 12:19:53 EET; 3h 46min ago
     Docs: https://docs.mongodb.org/manual
  Process: 27559 ExecStart=/usr/bin/mongod --auth --config /etc/mongod.conf (code=exited, status=1/FAILURE)
 Main PID: 27559 (code=exited, status=1/FAILURE)

systemd[1]: Started MongoDB Database Server.
systemd[1]: mongod.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mongod.service: Unit entered failed state.
systemd[1]: mongod.service: Failed with result 'exit-code'.

I have revised the mongod.conf file, which is located in /etc and includes the below:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
# engine: wiredTiger
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 23654
  bindIp: 0.0.0.0


# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

security:
  authorization: enabled

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options:

#auditLog:

#snmp:

Also the mongod.service located in /lib/systemd/system:

[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network.target

[Service]
User=mongodb
Group=mongodb
EnvironmentFile=-/etc/default/mongod
ExecStart=/usr/bin/mongod --auth --config /etc/mongod.conf
PIDFile=/var/run/mongodb/mongod.pid
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false

# Recommended limits for for mongod as specified in
# http://docs.mongodb.org/manual/reference/ulimit/#recommended-settings

[Install]
WantedBy=multi-user.target

I have also changed ownership using the below commands according to this post:

sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo chown mongodb:mongodb /tmp/mongodb-27017.sock
sudo chown mongodb:mongodb /tmp/mongodb-23654.sock
sudo chown -R mongodb:mongodb /var/log/mongodb

I have checked the folders and removed .lock files from /var/lib/mongodb/ which now looks like below:

 /var/lib/mongodb$ ls
    collection-0-2356690320700739995.wt   collection-2-2356690320700739995.wt  collection-8-5563536140699528517.wt  index-15-5563536140699528517.wt  index-5-5563536140699528517.wt  _mdb_catalog.wt    WiredTiger.wt
    collection-0-5563536140699528517.wt   collection-2-5563536140699528517.wt  diagnostic.data                      index-1-5563536140699528517.wt   index-6-2356690320700739995.wt  sizeStorer.wt
    collection-10-5563536140699528517.wt  collection-4-2356690320700739995.wt  index-10-2356690320700739995.wt      index-21-5563536140699528517.wt  index-7-5563536140699528517.wt  storage.bson
    collection-12-5563536140699528517.wt  collection-4-5563536140699528517.wt  index-11-5563536140699528517.wt      index-3-2356690320700739995.wt   index-9-2356690320700739995.wt  WiredTiger
    collection-14-5563536140699528517.wt  collection-6-5563536140699528517.wt  index-1-2356690320700739995.wt       index-3-5563536140699528517.wt   index-9-5563536140699528517.wt  WiredTigerLAS.wt
    collection-20-5563536140699528517.wt  collection-8-2356690320700739995.wt  index-13-5563536140699528517.wt      index-5-2356690320700739995.wt   journal                         WiredTiger.turtle

According to a previous post I had once enabled wiredTiger engine from mongod.conf but have now changed it back to the .conf shared above. Should I remove these files as well from this directory?

Finally, I have also checked the firewall for this port I am trying to use, it seems active. Have no idea what I should fix I have read lots of related posts but haven’t done yet…

chris · March 22, 2022, 11:28pm

What is in the log file?

What MongoDB version.
What OS, OS Version.

Konstantina_Arnaoutaki · March 23, 2022, 8:28am

The OS is Ubuntu16.04.7 LTS
Mongo db v4.2.0
Unfortunately the file /var/log/mongodb/mongod.log only includes logs two days ago, although I have just tried to start mongod again, I have also looked as syslog but nothing is in there too.

Ramachandra_Tummala · March 23, 2022, 10:44am

Is PIDfile matching in services file and mongod.cfg
Why to chmod temp sock files
Looks like you started mongod earlier on port 27017 and now on new port
Try to move these *.sock files and see if it comes up

Konstantina_Arnaoutaki · March 23, 2022, 12:22pm

You mean to remove both mongodb-23654.sock and mongodb-27017.sock?

> sudo rm /tmp/mongodb-23654.sock
> sudo rm /tmp/mongodb-27017.sock

Well the 27017 should have been created once I have changed for a while the mongod.conf

This kind of suggestions
sudo chown mongodb:mongodb /tmp/mongodb-27017.sock
were posted along with other suggested solutions in StackOverflow actually…

steevej · March 23, 2022, 12:46pm

Not always good.

and similar are used by a running process and you must not temper them with

What you need to do is:

Make sure no process is listening to the corresponding port with
ss -tlnp | grep 27017
Then if no process is listening you may safely remove with
sudo /bin/rm -f /tmp/mongodb-27017.sock

chris · March 23, 2022, 1:02pm

As the logging is not providing error information. I would suggest running mongod manually at this point.

sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf

This should error out. Given that nothing is appearing

Otherwise you’re basically randomly changing/removing files/configuration without cause.

Konstantina_Arnaoutaki · March 23, 2022, 3:40pm

I removed this

tmp/mongodb-27017.sock

There is only 23654. sock available now, still it didn’t work.

Konstantina_Arnaoutaki · March 23, 2022, 3:51pm

Nothing seems to be logged on the file

/var/lib/mongodb/mongodb.log

it is empty.
I tried also adding quotes as per the official instructions

so the path to log files looks like
path: "/var/log/mongodb/mongod.log"

still the same error…

steevej · March 23, 2022, 4:11pm

We need output of the following commands

ss -tlnp
ps -aef | grep [m]ongod
ls -l /tmp/mongodb-*

Is the log file

or

Konstantina_Arnaoutaki · March 23, 2022, 4:41pm

apologies for the misunderstanding, the correct file is

/var/log/mongodb/mongod.log

Result of

ss -tlnp
is

State       Recv-Q Send-Q                                                                         Local Address:Port                                                                                        Peer Address:Port
LISTEN      0      128                                                                                        *:26603                                                                                                  *:*                   users:(("node",pid=2711,fd=23))
LISTEN      0      128                                                                                        *:80                                                                                                     *:*
LISTEN      0      5                                                                                  127.0.1.1:53                                                                                                     *:*
LISTEN      0      128                                                                                        *:22                                                                                                     *:*
LISTEN      0      5                                                                                  127.0.0.1:631                                                                                                    *:*
LISTEN      0      128                                                                                        *:3000                                                                                                   *:*                   users:(("node",pid=2420,fd=23))
LISTEN      0      128                                                                                127.0.0.1:33980                                                                                                  *:*                   users:(("node",pid=2711,fd=22))
LISTEN      0      128                                                                                       :::80                                                                                                    :::*
LISTEN      0      128                                                                                       :::22                                                                                                    :::*
LISTEN      0      5                                                                                        ::1:631                                                                                                   :::*

and result of ls -l /tmp/mongodb-*
is
srwx------ 1 mongodb mongodb 0 Μάρ 23 17:24 /tmp/mongodb-23654.sock

The ps -aef | grep [m]ongod gave nothing.

steevej · March 23, 2022, 6:08pm

Okay.

Now remove /tmp/mongodb-… with

sudo /bin/rm -f /tmp/mongodb-23654.sock

Now manually start with

and post a screenshot of doing the above. Also post the content of

Konstantina_Arnaoutaki · March 23, 2022, 7:16pm

It still does not work and the /var/log/mongodb/mongod.log seems empty.

sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf
sudo vim /var/log/mongodb/mongod.log
sudo -u mongodb /usr/bin/mongod --auth --config /etc/mongod.conf
sudo systemctl status mongod
● mongod.service - MongoDB Database Server
>    Loaded: loaded (/lib/systemd/system/mongod.service; enabled; vendor preset: enabled)
>    Active: failed (Result: exit-code) since Τετ 2022-03-23 20:49:30 EET; 2min 56s ago
>      Docs: https://docs.mongodb.org/manual
>   Process: 27417 ExecStart=/usr/bin/mongod --auth --config /etc/mongod.conf (code=exited, status=1/FAILURE)
>  Main PID: 27417 (code=exited, status=1/FAILURE)
> 
 systemd[1]: Started MongoDB Database Server.
systemd[1]: mongod.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mongod.service: Unit entered failed state.
systemd[1]: mongod.service: Failed with result 'exit-code'.

I suppose the logs are saved somewhere else so tried to find where with the below, but I am not sure if it correctly used:

grep -r 'mongod.service: Main process exited, code=exited, status=1/FAILURE' *

Ramachandra_Tummala · March 24, 2022, 5:08am

Please show screenshot of the manual run
Did it complete successfully?
Can you run mongod as normal user on a different port,dbpath,logpath(not using your config file)

Konstantina_Arnaoutaki · March 24, 2022, 8:30am

The results when running the command you suggested:

Konstantina_Arnaoutaki · March 24, 2022, 8:30am

If I just try to mongod I get the below:

Ramachandra_Tummala · March 24, 2022, 8:49am

When you run mongod without any parameters it will try to start it on default port 27017 and default dirpath /data/db
Do you have this dir existing?

I asked you to run on different dbpath,logpath,port like this to rule out any issues with dirs
Something like below

Mongod --dbpath yourhomedir --logpath yourhomedir/mongod.log --port 29000 --fork

Once your mongod is up you can connect as
Mongo --port 29000

Konstantina_Arnaoutaki · March 24, 2022, 9:15am

The dir /data/db does exist

Konstantina_Arnaoutaki · March 24, 2022, 9:17am

Anyway I run this

mongod --dbpath /var/lib/mongodb --logpath /var/log/mongodb/mongod.log --port 29000 --fork

and get the below

Ramachandra_Tummala · March 24, 2022, 9:45am

Please use your home dir or someother dir where mongod can write.I specifically mentioned this
Do not use the dirs like /var/* as given in /etc/mongod.conf

Regarding /data/db error what is your os?
Your files are owned as root
They should be owned by mongod