RHEL9 systemctl gets permission error when database location moved or port changed

There is an issue with RHEL9 starting the mongod deamon with systemctl start mongod.

Two examples of this issue are:

  1. Moving the database location
  2. Changing Listing port

Example 1. Database location
storage:
dbPath: /var/lib/mongo

network interfaces

net:
port: 27017

systemctl start mongod – this works just fine as the defaults.

Now lets move the DB directory to \var\lib\mongo1 – SAME mount point, SAME permissions

[root@REDHAT-3212 firewalld]# ll /var/lib | grep mongo
drwxr-xr-x. 5 mongod mongod 4096 Apr 17 15:27 mongo
drwxr-xr-x. 2 mongod mongod 6 Apr 17 15:27 mongo1

storage:
dbPath: /var/lib/mongo1
systemctl start mongod –

FAILS with permission issue:

{“t”:{“$date”:“2024-04-17T15:30:16.960-05:00”},“s”:“E”, “c”:“WT”, “id”:22435, “ctx”:“initandlisten”,“msg”:“WiredTiger error message”,“attr”:{“error”:13,“message”:“[1713385816
:960404][159449:0x7f26d62a0b80], wiredtiger_open: [WT_VERB_DEFAULT][ERROR]: __posix_open_file, 815: /var/lib/mongo1/WiredTiger.lock: handle-open: open: Permission denied”}}

BUT – what if we SU to mongod and run it:
su mongod -c “mongod -f /etc/mongod.conf”

It WORKS just fine – Mongo daemon starts up just fine.
Its only an issuer when starting from systemctl

Example 2. Database port — lets change the port
storage:
dbPath: /var/lib/mongo

network interfaces

net:
port: 26019

When we start with systemctl it FAILS with BIND issue
systemctl start mongod –

{“t”:{“$date”:“2024-04-17T15:35:44.571-05:00”},“s”:“E”, “c”:“CONTROL”, “id”:20568,
“ctx”:“initandlisten”,“msg”:“Error setting up listener”,“attr”:{“error”:{“code”:9001,“codeName”:
“SocketException”,“errmsg”:“setup bind :: caused by :: Permission denied”}}}

But lets start it with SU
su mongod -c “mongod -f /etc/mongod.conf”

{“t”:{“$date”:“2024-04-17T15:36:52.742-05:00”},“s”:“I”, “c”:“NETWORK”, “id”:23016, “ctx”:“listener”,“msg”:“Waiting for connections”,“attr”:{“port”:26019,“ssl”:“off”}}
IT WORKS just fine

Why do we get permission issues with starting with systemctl – as a service it should start as mongod

When starting it with normal parameters we see the daemon running under mongod.
mongod 159959 1 29 15:38 ? 00:00:00 /usr/bin/mongod -f /etc/mongod.conf

It is hard to help you with your systemctl issue if you are not sharing the unit file that you are using.

The permission denied is on

so it would be nice to have the ll on the file.

One thing I notice is that on my laptop the user is mongodb not mongod like you seem to have. My installation is old since I mostly work out of Atlas so newer install may differ. May be you had a previous installation where mongodb was used so now obviously with mongod it would not work.

Or perhaps, you manually created your directories with mongod but the unit file uses mongodb user and group.

I cannot do a ls on the file because it is not created - this is a new directory that is empty.

This is installed on a RHEL9 server using the following as it pulls it from the latest repo.
There was nothing creating by hand, everything was done via the install - and yes, I
agree mongod vs mongodb is a bit weird – seems it install differently on different systems.

yum install -y mongodb-org

thanks

Since it may install differently, it is doubly important:

ok- I am not clear what the unit file you are requesting I post.
which file is this?
thanks

The unit file is the script used when you do systemctl start. See

We need to see the one for your mongod service.

found it - here it is

cat mongod.service

[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=mongod
Group=mongod
Environment=“OPTIONS=-f /etc/mongod.conf”
Environment=“MONGODB_CONFIG_OVERRIDE_NOFORK=1”
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/mongod $OPTIONS
RuntimeDirectory=mongodb

file size

LimitFSIZE=infinity

cpu time

LimitCPU=infinity

virtual memory size

LimitAS=infinity

open files

LimitNOFILE=64000

processes/threads

LimitNPROC=64000

locked memory

LimitMEMLOCK=infinity

total threads (user+kernel)

TasksMax=infinity
TasksAccounting=false

Recommended limits for mongod as specified in

https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=multi-user.target

Please review and update your posts:

It is very difficult to read when not formatted correctly and risks the post being ignored by some members of the community.

When MongoDB stops working on a RHEL system after a change from the default setting one of the first thing to check is whether or not SELinux is set to enforcing . Check this using the getenforce command.

If SELinux is set to enforcing then it can be set to permissive or can be completely disabled.

To change from the defaults AND keep SELinux in enforcing mode follow the SELinux Instructions from the manual.

https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-red-hat/#configure-selinux

Yes, it is SELINUX which is the culprit .

I found this: https://www.mongodb.com/docs/manual/tutorial/install-mongodb-on-red-hat/
Which states:
###################################################
If your MongoDB deployment uses custom settings for any of the following:

  • [MongoDB connection ports

  • [dbPath

You cannot use the MongoDB supplied SELinux policy. An alternative is to create a [custom SELinux policy]

###############################################

It gives instructions on how to build a new policy :slight_smile:

  1. Download the policy repository.

git clone htps://github.com/mongodb/mongodb-selinux

  1. Build the policy.
    cd mongodb-selinuxmake
    3. Apply the policy.
    sudo make install

However, even doing that does not fix the issue.
Even checking that the new policy is added for the new folder (/data/mongo)

/data/mongo(/.*)?                              all files          system_u:object_r:mongod_var_lib_t:s0
/data/mongo.*                                      all files          system_u:object_r:mongod_var_lib_t:s0
/run/mongodb(/.*)?                                 all files          system_u:object_r:mongod_runtime_t:s0
/usr/bin/mongod                                    regular file       system_u:object_r:mongod_exec_t:s0
/var/lib/mongo(/.*)?                               all files          system_u:object_r:mongod_var_lib_t:s0
/var/log/mongodb(/.*)?                             all files          system_u:object_r:mongod_log_t:s0

We still get permissioned denied when starting up with systemctl.

It is irritating that its documented that it needs a fix, provides a fix, yet still does not work.

I can of course turn off SELINUX but not sure that is the best solution.

Thanks for the help

1 Like

Thanks to you for digging up the information.

With

you might need to add

/var/lib/mongo1(/.*)?                               all files          system_u:object_r:mongod_var_lib_t:s0

And please as mentioned

I moved it to /data/mongo and you can see in semodule it is added

/data/mongo(/.*)?

Same permission issue - ugh

Since you moved things around you would need to share the whole configuration again. This is the only way to see possible mismatch.

We need the unit file, the mongodb configuration file, the command you used to start, the selinux config, the ls -ld of all directories involved.