HTTP: Error receiving data -- Connection reset by peer

jtvdw · February 14, 2020, 11:32am

Hi,

Since I’ve rebooted my server yesterday I’m getting this error:

mongodb’ failed protocol test [HTTP] at [localhost]:27017/ [TCP/IP] – HTTP: Error receiving data – Connection reset by peer

‘mongodb’ start: ‘/usr/bin/sudo mongod --dbpath=/var/lib/mongodb’

Tried a couple of thing but no luck.

Also checked with curl -v localhost:27017 and output below:

Rebuilt URL to: localhost:27017/
Trying ::1…
TCP_NODELAY set
connect to ::1 port 27017 failed: Connection refused
Trying 127.0.0.1…
TCP_NODELAY set
Connected to localhost (127.0.0.1) port 27017 (#0)

GET / HTTP/1.1
Host: localhost:27017
User-Agent: curl/7.58.0
Accept: /

Recv failure: Connection reset by peer
stopped the pause stream!
Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Any input or help will be much appreciated.

Thanks.

Stennie_X · February 15, 2020, 4:46am

What O/S version are you running and how do you normally start your MongoDB service? Was this a clean reboot or an unexpected/forced restart?

The error messages suggest you are trying to connect to localhost:27017 but the MongoDB service isn’t running. If this coincides with rebooting your server, the MongoDB service may not be set to run on startup or has perhaps encountered an error. I would start by checking the MongoDB server logs.

Is this output from monit? It is unusual to do an HTTP check for mongod on the default port since the server uses the MongoDB Wire Protocol, but if the server is accessible the result for an HTTP request should be similar to: “It looks like you are trying to access MongoDB over HTTP on the native driver port.”.

Regards,
Stennie

jtvdw · February 16, 2020, 9:09am

Currently running Ubuntu 18 and it looks like it’s not set to start manually, I took over from the previous person. It was a unexpected/forced restart.

Yes, that’s correct but the mongodb service is down, killed the pid and tried starting the service but still fails, should I post the logs here as well?

Yes, it’s uning monit Also not sure why it was setup that way, like I mentioned I took over from anoteher guy.

What should I change or do to fix this issue.

Much appreciated.

jtvdw · February 17, 2020, 8:34pm

Hi,

I followed this link but still no go for me

Any help would be appreciated.

Thanks.

Stennie_X · February 18, 2020, 4:57am

You could post relevant lines from the logs if it isn’t obvious why the MongoDB process isn’t starting successfully.

What command line are you using to start the MongoDB service? If the installation used the official MongoDB packages you should be able to use systemctl: Start MongoDB.

The default log file location is /var/log/mongodb/mongod.log. If the mongod process is unable to start successfully, the last log lines should provide some indication of the reason (for example file permissions or integrity problems).

Regards,
Stennie

jtvdw · February 19, 2020, 6:55am

Here is the lst lines of the log file: (Path : cat /var/log/mongodb/mongodb.log)
2020-02-19T08:47:17.620+0200 I CONTROL [main] ***** SERVER RESTARTED *****
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] MongoDB starting : pid=25072 port=27017 dbpath=/var/lib/mongodb 64-bit host=dis002-truservcomm-jhb1-001
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] db version v3.6.3
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] allocator: tcmalloc
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] modules: none
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] build environment:
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] distarch: x86_64
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] target_arch: x86_64
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] options: { config: “/etc/mongodb.conf”, net: { bindIp: “127.0.0.1”, port: 27017, unixDomainSocket: { pathPrefix: “/run/mongodb” } }, security: { authorization: “disabled” }, setParameter: { enableLocalhostAuthBypass: “0” }, storage: { dbPath: “/var/lib/mongodb”, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, path: “/var/log/mongodb/mongodb.log” } }
2020-02-19T08:47:17.629+0200 I STORAGE [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /var/lib/mongodb, terminating
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] now exiting
2020-02-19T08:47:17.629+0200 I CONTROL [initandlisten] shutting down with code:100

I’m using this command : sudo service mongodb start (or status) but still fails… This is the output of the command you mentioned:
user@server:~$ sudo systemctl start mongodb (Works)
user@server:~$ sudo systemctl start mongod
Failed to start mongod.service: Unit mongod.service not found.

Weird, had a look at the log file location but this is the output:
sudo tail -f /var/log/mongodb/mongod.log
tail: cannot open ‘/var/log/mongodb/mongod.log’ for reading: No such file or directory

Please let me know if you need more details as well.

Much appreciated.

Stennie_X · February 19, 2020, 9:05am

You should plan to install the latest minor release (currently 3.6.17) as there have been a lot of improvements and stability fixes. A minor release in the same series (3.6.x) does not introduce any compatibility changes.

You need to fix the permissions on /var/lib/mongodb to match the user you are running mongod as.

For example, assuming mongodb user and group: sudo chmod -R mongodb:mongodb /var/lib/mongodb.

If service works, that’s fine. systemctl is the equivalent of the service command for systemd which is the usual service manager for Ubuntu 18.

Given the older server version (3.6.3) and that you are starting the service using mongodb (rather than mongod) I suspect you have installed Ubuntu’s MongoDB package rather than the officially maintained MongoDB package for Ubuntu.

There are some differences in default configuration values between the two packages (such as service name and default paths. I’d recommend using the version from MongoDB as it is more actively maintained and in sync with the server documentation, but you can plan that change after you get your service running successfully.

Regards,
Stennie

jtvdw · February 19, 2020, 10:33am

Okay, I will update mongodb to the latest version.

I’ve done this now and mongodb service still fails. In the log is only entries from 2019/12 nothing to date.

I’ve ran the “sudo service mongodb start” command, not failing but service still down. Removed the mongodb.lock file as well.

I really have a battle with this, but thank for your help so far.

Let me know what other details you need from me.

Much appreciated.

chris · February 19, 2020, 12:04pm

Having a sample of the logs like in the previous post is about the only way to give meaningful help. For example it was very obvious there was/is a permissions error on the data directory.

jtvdw · February 19, 2020, 12:08pm

2020-02-19T14:01:59.251+0200 I CONTROL [main] ***** SERVER RESTARTED *****
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] MongoDB starting : pid=22521 port=27017 dbpath=/var/lib/mongodb 64-bit host=dis002-truservcomm-jhb1-001
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] db version v3.6.3
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] git version: 9586e557d54ef70f9ca4b43c26892cd55257e1a5
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] allocator: tcmalloc
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] modules: none
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] build environment:
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] distarch: x86_64
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] target_arch: x86_64
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] options: { config: “/etc/mongodb.conf”, net: { bindIp: “127.0.0.1”, port: 27017 }, security: { authorization: “disabled” }, setParameter: { enableLocalhostAuthBypass: “0” }, storage: { dbPath: “/var/lib/mongodb”, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, path: “/var/log/mongodb/mongodb.log” } }
2020-02-19T14:01:59.259+0200 I STORAGE [initandlisten] exception in initAndListen: Location28596: Unable to determine status of lock file in the data directory /var/lib/mongodb: boost::filesystem::status: Permission denied: “/var/lib/mongodb/mongod.lock”, terminating
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] now exiting
2020-02-19T14:01:59.259+0200 I CONTROL [initandlisten] shutting down with code:100

I tried to upgrade from 3.6.3 to 3.6.17 but I broke it now, this is the error message I got:

sudo systemctl status mongodb

mongodb.service - LSB: An object/document-oriented database
Loaded: loaded (/etc/init.d/mongodb; generated)
Active: failed (Result: exit-code) since Wed 2020-02-19 14:02:00 SAST; 5min ago
Docs: man:systemd-sysv-generator(8)
Process: 22500 ExecStart=/etc/init.d/mongodb start (code=exited, status=1/FAILURE)
Main PID: 11010 (code=exited, status=100)

Feb 19 14:01:59 dis002-truservcomm-jhb1-001 systemd[1]: Starting LSB: An object/document-oriented database…
Feb 19 14:01:59 dis002-truservcomm-jhb1-001 mongodb[22500]: * Starting database mongodb
Feb 19 14:02:00 dis002-truservcomm-jhb1-001 mongodb[22500]: …fail!
Feb 19 14:02:00 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Control process exited, code=exited status=1
Feb 19 14:02:00 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.
Feb 19 14:02:00 dis002-truservcomm-jhb1-001 systemd[1]: Failed to start LSB: An object/document-oriented database.

Also if I try to “sudo apt install -y mongodb” I get this:

Reading package lists… Done
Building dependency tree
Reading state information… Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
mongodb-org-shell : Conflicts: mongodb but 1:3.6.3-0ubuntu1.1 is to be installed
Conflicts: mongodb-clients but 1:3.6.3-0ubuntu1.1 is to be installed
Conflicts: mongodb-server but 1:3.6.3-0ubuntu1.1 is to be installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.

Any help would be appreciated.

Thanks.

chris · February 19, 2020, 12:15pm

You still have a permission issue with the files/folders. Assuming mongodb as the uid/guid make sure you run @Stennie_X’s chown command extra ateention to the -R

For upgrading from ubuntu to mongodb-org-* you cannot simply add the repo and install. You will have to remove the Ubuntu version(s). I’ll see if there is a relevant doc somewhere. Note: Make sure this is not the only copy. You should have a backup or other running healthy replicas.

jtvdw · February 19, 2020, 12:18pm

I have run it -R but keeps changes back, I’m a user and not root, sudo the whole time, could that be an issue?

chris · February 19, 2020, 12:18pm

This should be of some help:

jtvdw · February 19, 2020, 12:19pm

Looks like I fixed the error but still on 3.6.3 but when I try starting the service is still fails:

sudo service mongodb status
mongodb.service - An object/document-oriented database
Loaded: loaded (/lib/systemd/system/mongodb.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2020-02-19 14:16:52 SAST; 3s ago
Docs: man:mongod(1)
Process: 27493 ExecStart=/usr/bin/mongod --unixSocketPrefix={SOCKETPATH} --config {CONF} $DAEMON_OPTS (code=exited, status=48)
Main PID: 27493 (code=exited, status=48)

Feb 19 14:16:52 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:16:52 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=48/n/a
Feb 19 14:16:52 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.

jtvdw · February 19, 2020, 12:21pm

This is the last output of the mongodb directory:

drwx------ 2 mongodb mongodb 4096 Feb 19 13:42 journal/
-rw------- 1 mongodb mongodb 36864 Feb 19 13:41 _mdb_catalog.wt
-rw------- 1 mongodb mongodb 0 Feb 19 14:16 mongod.lock
-rw------- 1 mongodb mongodb 36864 Feb 19 13:42 sizeStorer.wt
-rw------- 1 mongodb mongodb 114 Feb 14 14:42 storage.bson
-rw------- 1 mongodb mongodb 48 Feb 14 14:42 WiredTiger
-rw------- 1 mongodb mongodb 4096 Feb 19 13:41 WiredTigerLAS.wt
-rw------- 1 mongodb mongodb 21 Feb 14 14:42 WiredTiger.lock
-rw------- 1 mongodb mongodb 1057 Feb 19 13:42 WiredTiger.turtle
-rw------- 1 mongodb mongodb 217088 Feb 19 13:42 WiredTiger.wt

chris · February 19, 2020, 12:23pm

48 A newly started mongod or mongos could not start listening for incoming connections, due to an error.

More is probably in logs or via journalctl -u mongod

jtvdw · February 19, 2020, 12:28pm

Ran the command with output:

sudo journalctl -u mongod
– Logs begin at Tue 2019-07-16 21:43:12 SAST, end at Wed 2020-02-19 14:27:28 SAST. –
– No entries –

chris · February 19, 2020, 12:32pm

Boo. Regular log file ?

jtvdw · February 19, 2020, 12:36pm

tail -f /var/log/mongodb/mongodb.log
2020-02-19T14:29:16.596+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.1d 10 Sep 2019
2020-02-19T14:29:16.596+0200 I CONTROL [initandlisten] allocator: tcmalloc
2020-02-19T14:29:16.596+0200 I CONTROL [initandlisten] modules: none
2020-02-19T14:29:16.596+0200 I CONTROL [initandlisten] build environment:
2020-02-19T14:29:16.596+0200 I CONTROL [initandlisten] distarch: x86_64
2020-02-19T14:29:16.596+0200 I CONTROL [initandlisten] target_arch: x86_64
2020-02-19T14:29:16.596+0200 I CONTROL [initandlisten] options: { config: “/etc/mongodb.conf”, net: { bindIp: “127.0.0.1”, port: 27017, unixDomainSocket: { pathPrefix: “/run/mongodb” } }, security: { authorization: “disabled” }, setParameter: { enableLocalhostAuthBypass: “0” }, storage: { dbPath: “/var/lib/mongodb”, journal: { enabled: true } }, systemLog: { destination: “file”, logAppend: true, path: “/var/log/mongodb/mongodb.log” } }
2020-02-19T14:29:16.596+0200 E STORAGE [initandlisten] Failed to set up listener: SocketException: Address already in use
2020-02-19T14:29:16.596+0200 I CONTROL [initandlisten] now exiting
2020-02-19T14:29:16.596+0200 I CONTROL [initandlisten] shutting down with code:48

Looks the same as previous log file with error code 48 …

jtvdw · February 19, 2020, 12:39pm

Ran the command “sudo journalctl -u mongodb” with output below:

Feb 19 14:12:46 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:12:46 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=100/n/a
Feb 19 14:12:46 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.
Feb 19 14:13:06 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:13:06 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=100/n/a
Feb 19 14:13:06 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.
Feb 19 14:14:12 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:14:12 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=100/n/a
Feb 19 14:14:12 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.
Feb 19 14:15:49 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:15:49 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=100/n/a
Feb 19 14:15:49 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.
Feb 19 14:16:52 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:16:52 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=48/n/a
Feb 19 14:16:52 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.
Feb 19 14:21:41 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:21:41 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=48/n/a
Feb 19 14:21:41 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.
Feb 19 14:24:21 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:24:21 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=48/n/a
Feb 19 14:24:21 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.
Feb 19 14:29:16 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:29:16 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=48/n/a
Feb 19 14:29:16 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.
Feb 19 14:37:08 dis002-truservcomm-jhb1-001 systemd[1]: Started An object/document-oriented database.
Feb 19 14:37:08 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Main process exited, code=exited, status=48/n/a
Feb 19 14:37:08 dis002-truservcomm-jhb1-001 systemd[1]: mongodb.service: Failed with result ‘exit-code’.