Timeout error after database directory change

Ops and Admin
1

There are 3 storages on the computer:

  • SSD (very small)
  • HDD (hereinafter it will be referred to as /mnt/V1)
  • HDD (hereinafter it will be referred to as /mnt/V2)

By default, the databases were stored in the SSD, but the free space there ran out. I decided that the data should be accumulated in the /mnt/V2/ and I edited /etc/mongod.conf:

storage:
dbPath: /mnt/V2/mongodb

systemLog:
path: /mnt/V2/mongod.log

Then I created the appropriate paths and corrected the permissions for them:

sudo chown $USER -R /mnt/V2/mongodb
sudo chown $USER -R /mnt/V2/mongod.log

Then I rebooted the OS.

An error occurs after an attempt to connect to DB (client[db_name]):

pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 608eb8fa091e8bf0c343b405, topology_type: Single, servers: [<ServerDescription ('localhost', 27017) server_type: Unknown, rtt: None, error=AutoReconnect('localhost:27017: [Errno 111] Connection refused')>]>

There was no error before changing the database directory.

2

Looks like mongod server did not restart.

Share output of ps -aef | grep [m]ongo.

Share the content of /mnt/V2/mongod.log if present. Since the log file is directly in /mnt/V2/ then mongod should have write permission in order to create and rotate the log file.

3

ps -aef | grep mongo
platon 2019 1481 0 15:26 pts/0 00:00:00 grep --color=auto mongo

mongod.log:

{"t":{"$date":"2021-05-02T13:37:28.770+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2021-05-02T13:37:28.819+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2021-05-02T13:37:28.819+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2021-05-02T13:37:28.820+00:00"},"s":"I",  "c":"STORAGE",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1055,"port":27017,"dbPath":"/mnt/V2/mongodb","architecture":"64-bit","host":"platon"}}
{"t":{"$date":"2021-05-02T13:37:28.820+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.5","gitVersion":"ff5cb77101b052fa02da43b8538093486cf9b3f7","openSSLVersion":"OpenSSL 1.1.1f  31 Mar 2020","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2004","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2021-05-02T13:37:28.820+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"20.04"}}}
{"t":{"$date":"2021-05-02T13:37:28.820+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"127.0.0.1","port":27017},"processManagement":{"timeZoneInfo":"/usr/share/zoneinfo"},"storage":{"dbPath":"/mnt/V2/mongodb","journal":{"enabled":true}},"systemLog":{"destination":"file","logAppend":true,"path":"/mnt/V2/mongod.log"}}}}
{"t":{"$date":"2021-05-02T13:37:28.857+00:00"},"s":"E",  "c":"STORAGE",  "id":20557,   "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"IllegalOperation: Attempted to create a lock file on a read-only directory: /mnt/V2/mongodb"}}
{"t":{"$date":"2021-05-02T13:37:28.857+00:00"},"s":"I",  "c":"REPL",     "id":4784900, "ctx":"initandlisten","msg":"Stepping down the ReplicationCoordinator for shutdown","attr":{"waitTimeMillis":10000}}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"COMMAND",  "id":4784901, "ctx":"initandlisten","msg":"Shutting down the MirrorMaestro"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"SHARDING", "id":4784902, "ctx":"initandlisten","msg":"Shutting down the WaitForMajorityService"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"NETWORK",  "id":20562,   "ctx":"initandlisten","msg":"Shutdown: going to close listening sockets"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"NETWORK",  "id":4784905, "ctx":"initandlisten","msg":"Shutting down the global connection pool"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"STORAGE",  "id":4784906, "ctx":"initandlisten","msg":"Shutting down the FlowControlTicketholder"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"-",        "id":20520,   "ctx":"initandlisten","msg":"Stopping further Flow Control ticket acquisitions."}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"NETWORK",  "id":4784918, "ctx":"initandlisten","msg":"Shutting down the ReplicaSetMonitor"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"SHARDING", "id":4784921, "ctx":"initandlisten","msg":"Shutting down the MigrationUtilExecutor"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"CONTROL",  "id":4784925, "ctx":"initandlisten","msg":"Shutting down free monitoring"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"STORAGE",  "id":4784927, "ctx":"initandlisten","msg":"Shutting down the HealthLog"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"STORAGE",  "id":4784929, "ctx":"initandlisten","msg":"Acquiring the global lock for shutdown"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"-",        "id":4784931, "ctx":"initandlisten","msg":"Dropping the scope cache for shutdown"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"FTDC",     "id":4784926, "ctx":"initandlisten","msg":"Shutting down full-time data capture"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"initandlisten","msg":"Now exiting"}
{"t":{"$date":"2021-05-02T13:37:28.863+00:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"initandlisten","msg":"Shutting down","attr":{"exitCode":100}}
4

The server mongod could not restart because:

I suspect that $USER was not defined correctly when you chown. The directory /mnt/V2/mongodb must be writable by the user running mongod. On my CentOS the user is also named mongod.

PS: I asked for grep [m]ongo with the brackets in order to not see the command grep itself in the output. See the difference below:

: steevej @ xps ; ps -aef | grep mongod
mongod    7799     1 11 12:23 ?        00:00:00 /usr/bin/mongod -f /etc/mongod.conf
steevej   7847  4992  0 12:23 pts/0    00:00:00 grep mongod
: steevej @ xps ; ps -aef | grep [m]ongod
mongod    7799     1  1 12:23 ?        00:00:01 /usr/bin/mongod -f /etc/mongod.conf
5

Okay, I have specified user name explicitly:
sudo chown platon -R /mnt/V2/mongodb
sudo chown platon -R /mnt/V2/mongod.log
That didn’t help.

About user permissions:
groups platon
platon : platon adm cdrom sudo dip plugdev lxd docker

Grep with [] does not print any results, so I applied the command without them.

6

No output means no mongo process running.

Most likely because platon is not the appropriate user. You have to look at the systemd configuration for mongod.

7

But why did it work before changing the DB directory?

8

May be because the old directory has been created correctly and the new one, not.

9

Can anyone provide a step-by-step guide to solving the problem?

10

Have you solved the problem finally? I meet the same problem after changing the application directory.

11

Please start a new thread with your issue and provide more details.

Unless you really have the same problem. In this case the solution would be the same. Fix the directory permissions by making sure the user (should be mongodb) with which you start mongod can write into it.

1 Like
12
  1. sudo nano /etc/mongod.conf 
storage:
dbPath: /mnt/VOLUME_NAME/mongodb

systemLog:
path: /mnt/VOLUME_NAME/mongod.log

CTRL+S
CTRL+X

mkdir /mnt/VOLUME_NAME/mongodb
touch /mnt/VOLUME_NAME/mongod.log
sudo chown mongodb -R /mnt/VOLUME_NAME/mongodb
sudo chown mongodb -R /mnt/VOLUME_NAME/mongod.log
13

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.