My process won't start using systemctl but starts using commandLine

Hello,

I tried to deploy a replicaSet in my development environment, but I’m facing a weird problem in my second server.

In my first server, I’m using a Debian 11 and in my second server I’m using Rocky Linux 9.3

This is the conf file from my first server:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
  engine: wiredTiger
  wiredTiger:
    collectionConfig:
      blockCompressor: zstd

# where to write logging data.
systemLog:
  destination: file
  logAppend: false
  logRotate: rename
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIpAll: true

# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security:
security:
  authorization: enabled
  keyFile: /var/lib/mongodb/file.key

#replication:
replication:
  replSetName: devReplicaSet

And this is the conf file from my second server:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
  engine: wiredTiger
  wiredTiger:
    collectionConfig:
      blockCompressor: zstd

# where to write logging data.
systemLog:
  destination: file
  logAppend: false
  logRotate: rename
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0

# how the process runs
processManagement:
  timeZoneInfo: /usr/share/zoneinfo

#security
security:
  authorization: enabled
  keyFile: /var/lib/mongodb/file.key
  transitionToAuth: true

#replication:
replication:
  replSetName: devReplicaSet

Both of my keyFiles are with the recommend permissions (chown and chmod):

jose_rocha@lnx-mongodb-dev-001:~$ ls -hla /var/lib/mongodb/file.key
-r-------- 1 mongodb mongodb 1.0K Nov 26 04:26 /var/lib/mongodb/file.key
[jose_rocha@lnx-mongodb-dev-002 ~]$ ls -hla /var/lib/mongodb/file.key
-r--------. 1 mongod mongod 1.0K Nov 26 04:27 /var/lib/mongodb/file.key

Now the weird issue. When I try to start my mongod process at second server, I receive the following error in mongod.log

{"t":{"$date":"2023-11-28T13:47:13.588-03:00"},"s":"I",  "c":"ACCESS",   "id":20254,   "ctx":"main","msg":"Read security file failed","attr":{"error":{"code":30,"codeName":"InvalidPath","errmsg":"error opening file: /var/lib/mongodb/file.key: bad file"}}}
{"t":{"$date":"2023-11-28T13:47:13.588-03:00"},"s":"D1", "c":"ASSERT",   "id":23074,   "ctx":"main","msg":"User assertion","attr":{"error":"Location5579201: Unable to acquire security key[s]","file":"src/mongo/db/auth/authorization_manager_global.cpp","line":74}}

And the process won’t start, but if I try to start the mongod process using the commandLine, everything work fine, including the replicaSet:

[jose_rocha@lnx-mongodb-dev-002 ~]$ sudo -u mongod /usr/bin/mongod -f /etc/mongod.conf

I’m completely lost here. I sincerely don’t know what else I could look. Any help ?

EDIT
This is the content of mongod.service from the second server:

[Unit]
Description=MongoDB Database Server
Documentation=https://docs.mongodb.org/manual
After=network-online.target
Wants=network-online.target

[Service]
User=mongod
Group=mongod
Environment="OPTIONS=-f /etc/mongod.conf"
Environment="MONGODB_CONFIG_OVERRIDE_NOFORK=1"
EnvironmentFile=-/etc/sysconfig/mongod
ExecStart=/usr/bin/mongod $OPTIONS
RuntimeDirectory=mongodb
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false
# Recommended limits for mongod as specified in
# https://docs.mongodb.com/manual/reference/ulimit/#recommended-ulimit-settings

[Install]
WantedBy=multi-user.target

Hi @Jose_Felipe_Goncalves_Rocha,
Why only 2 servers?
Dumb question, is the generated keyfile the same for both servers?
In case the answer is yes, could you try to regenerate it maybe from server 2 or server 1 and redistribute it?

https://www.mongodb.com/docs/v6.2/tutorial/deploy-replica-set-with-keyfile-access-control/#:~:text=openssl%20rand%20-base64%20756,400%20<path-to-keyfile>

I personally prefer to create a replica set with 3 nodes that have the same os and characteristics for convenience.

Regards