Measure ssh transfer speed of mongo db data?

Working with Data
1

Hello,

when tunnelling into my mongo db via ssh port forwarding, I get quite slow data transfer response. I am using time series collections.

I measured the time it takes to query 276 Million data points (about 1.2 Million 5-second bins of data in the time series db)…

  • … when I query from an external pc that is connected to the mongodb via ssh:
    about 18 minutes (with huge variance: sometimes 12 minutes, sometimes 22, etc…)
  • … when I query directly on the server where the mongo db is located (i.e. localhost):
    about 1min 40s

So, there definitely seems to be a huge overhead due to the remote data transfer, while the mongo db is actually working ok in terms of the actual query speed.

What I’d like to ask:

  • How to approximately compute the size of the 276 Million data points (float64) when transferred via wifi ssh?
  • How to measure the speed of data transfer that my PC’s are using? (Maybe the router or the pc’s wifi adapter themselves limit the transfer too much).
  • Is there a way to compress the data before the transfer somehow or, in general, make the remote query more efficient?

Thanks!

3

Any reasons why you encrypt via SSH tunnel rather than TLS?

I suspect TLS will be more efficient as there is 2 extra steps with SSH. With SSH your data is sent from the server to its local SSHd to get encrypted and then sent to the client SSHd to get decrypted before being sent to the client driver. These extra steps might have a big influence on performance with 276 million data points.

But to effectively test the overhead of SSH is to perform your queries on the PC with and then without SSH. Your localhost test is useless as a comparison point with your remote PC.

4

Hey Steeve,

thanks for your reply. Can you give me some buzzwords on how to set up TLS?

The comparison I made includes the overhead caused by SSH + transferring the data between the PC’s via WiFi. So, what you are suggesting is to disentangle the two in a test, in order to find out the overhead JUST caused by SSH, right?

Thanks!

5

Start with https://www.mongodb.com/docs/manual/tutorial/configure-ssl/

yes

6

Hello Steeve,

I finally got the time to try switching to TLS.
So, firstly what I did was editing the net settings in the config file like this:

    tls: 
       mode: allowTLS
       certificateKeyFile: PATH_TO_MONGO\bin\mongod-cert.pem
       allowConnectionsWithoutCertificates: true
       allowInvalidCertificates: true
       allowInvalidHostnames: true

I use the minimum security settings and a self-signed certificate for now, for two reasons:
(i) I just want to test whether it works at the minimum settings, then, later on, increase the security level, again
(ii) I am connecting to the mongodb within a private network, and I am not planning to use it to serve external clients, just my own pc inside the network. Therefore, I guess, it is ok to use a self-signed certificate.

On my host pc for the mongodb, I managed to connect to the mongodb like this from python:

import pymongo 
client = pymongo.MongoClient('mongodb://localhost:27017/',tls=True,tlsCAfile="PATH_TO_MONGO\\cert.pem")

I tested the connection and it worked fine.

Next, I tried to connect to my mongodb from another pc (the “client”) inside the same private router network. This always gives me the following error, when trying to request data from the host:

ServerSelectionTimeoutError: 192.168.X.XXX:27017: timed out, Timeout: 30s, Topology Description: 
<TopologyDescription id: 62f22029b65b0358c6e03ca4, topology_type: Single, servers: 
[<ServerDescription ('192.168.X.XXX', 27017) server_type: Unknown, rtt: None, 
error=NetworkTimeout('192.168.X.XXX:27017: timed out')>]>

Apparently, the client cannot connect to the hosted mongo db in time. The remaining mongo conf settings that I use are:

# where and how to store data.
storage:
  dbPath: D:\MongoDB\Server\5.0\data
  journal:
    enabled: true

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path:  PATH_TO_LOG\mongod.log
  quiet: true

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1, localhost, 192.168.X.XXX

I tried different combinations of the IPs listed in bindip, including adding 0.0.0.0. Nothing prevents the error from occurring. There must be something I have done wrong.

Usually, as stated above, I connect to the db via SSH port forwarding. So, on my client ubuntu machine, I start a terminal forwarding the host’s 27017 port to the client’s 8000 port:

ssh -N -L 8000:192.168.X.XXX:27017 user@192.168.X.XXX

On the client, once the tunnel is up, I connect like this:

client = pymongo.MongoClient('mongodb://localhost:8000/')

This also now works with the tls option enabled, as in the python code above, but only when the SSH tunnel is up (of course). However, I think there is no point in using TLS via the SSH tunnel, because the speed of transfer will still be limited by the SSH tunnel, right? Therefore, I am trying to now connect directly without the SSH tunnel, which however gives the timeout error.

Can you help me with this? Any ideas?

Thank you!

Best, JZ

7

Is the certificate available on the other PC?

Share the python code you use on the other PC when it failed with

8

I found that it is not a certificate or TLS related issue, because the timeout error also occurs without it. In that case the net settings look like that:

 net:
  port: 27017
  bindIp: 127.0.0.1, 0.0.0.0

I also tried it with only 0.0.0.0.

The python code when trying to connect directly without SSH is this:

uri_mongo = 'mongodb://192.168.X.XXX:27017/'
client = pymongo.MongoClient(uri_mongo)
db = client['db_name']
db.list_collection_names()

This results in the timeout error.

For SSH, I forward the port 27017 of the db host to port 8000 of the client and can run the code above successfully, when replacing uri_mongo with:

uri_mongo = 'mongodb://localhost:8000/'

My client machine runs on Ubuntu 22.04 LTS and my PC on Windows 11. I thought I might have to open the Windows Firewall. I added in- and outbound permission rules for mongo.exe, mongos.exe, mongod.exe and the port 27017 TCP in general. It still does not work. I am out of ammo at this point.

Thanks! Best, JZ

9

Hello,

I solved the problem. It was indeed a windows firewall problem with the port settings, and following these steps in the mongo docs solved it:

2 Likes
11

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.