70 MB of daily traffic with no interaction

MongoDB Atlas
1

I have a very simple node js application which connects to a free shared MongoDB. Looking at the metrics of the server I can see a daily 60 MB inbound and 10 MB outbound traffic to the shared instance. This traffic occurs even if there is no interaction with the database.
Can anyone explain why this happens? Is this intended? Is there any way to avoid this constant traffic?

Just for reference and why this can be a problem. The application is hosted on an EC2 instance within a AWS Free Tier account and on the same region as the database server. The AWS Free Tier includes 1 GB of regional traffic transfer, that is, traffic within the same region but across availability zoned (data centres). It turns out that the shared MongoDB is hosted in the same region but in a different availability zone. So with 70 MB of daily traffic I exceed the 1 GB monthly limit pretty fast.

2

Hi @T3rm1,

I assume you’re talking about a M0 free-tier Atlas cluster in this scenario but please correct me if I am wrong. Please contact the Atlas in-app chat support regarding the network usage question as they’ll have more insight into your cluster.

In saying the above, you could also try to remove all network access list entries and ensure connections are at 0 and monitor to see if anything might be coming from the application side (perhaps unexpected operations) as a troubleshooting step.

Regards,
Jason

3

Ok, I tried that but I got in contact with a person who didn’t really understand the issue. In the end he said that this behaviour is normal.
I think this is really bad design then. 2.1 GB of traffic each month on an idle connection is too much.

4

I believe the behaviour (network traffic shown on the Network metrics chart in the Atlas UI) you’ve mentioned in specific reference to the M0 tier cluster is expected assuming you are not passing through any operations to the server from your application. Firstly, a few points:

  • Replica set needs constant communication with each other to ensure the correct working order of the set, namely to provide high availability in the event of network or hardware issues
  • The driver also constantly communicates with all voting members of a replica set to ensure quick recovery in service, and to monitor the overall health of the replica set
  • Thus, an idle database is never really idle, especially in a distributed system. This is the nature of the system. Constant communication is a key component of the overall system
  • I cannot comment about AWS free tier allowances, but have you ever hit the 10GB ingoing / outgoing network limit in your free tier cluster? Shared clusters are generally used as economical clusters for getting started with MongoDB and for low-throughput applications.

My guess in this scenario (again, assuming you are not performing any operations from your application), is that the “idle” connection(s) still need to communicate with the cluster which generally involves a series of checks. Please see further details on the server monitoring specs documentation.

I’m going to use the heartbeatFrequencyMS option in the pymongo driver as an example on my test M0 cluster to help demonstrate.

In the following screenshots, you will see a single client connect at 2 different points for several minutes each time. The first block where the connections spike will be when the heartbeatFrequencyMS value is set to 100000 - Note the Network metrics during this time. After close()-ing the connection and waiting several minutes, another connection is made again but with a heartbeatFrequencyMS value set to 20000 (20% of the first connection) in which you can see higher network usage:

image
image1522×1102 41.5 KB

For reference, I executed this code in a test python environment using the pymongo driver:

>>> client = pymongo.MongoClient(uri, heartbeatFrequencyMS=100000, tlsCAFile=certifi.where())
>>> client.close()

/// Waited several minutes here

>>> client = pymongo.MongoClient(uri, heartbeatFrequencyMS=20000, tlsCAFile=certifi.where())
>>> client.close()

Note: Not all drivers behave exactly the same and the above is only for demonstration purposes regarding the network metric on a driver connection performing CRUD operations.

It’s important to note that I am not advising you to change the heartbeatFrequencyMS option here. This is only to demonstrate that even though my connection is not performing any read or write operations, the connection(s) still requires network usage.

I would go over the monitoring spec documentation linked earlier in my comment as I believe this would be useful here.

Regards,
Jason

1 Like
5

Thanks a lot for your effort.

  • The driver also constantly communicates with all voting members of a replica set to ensure quick recovery in service, and to monitor the overall health of the replica set

In that case this must be the cause. Still I think 60 MB of traffic within 24 hours on a 100% idle connection (no commands sent) is huge.

  • Thus, an idle database is never really idle, especially in a distributed system. This is the nature of the system. Constant communication is a key component of the overall system

I’m by no means an expert but that doesn’t sound right. There is no need to have constant communication with replica servers that are not used at all. This should only happen if there is some kind of a problem with the connection to the main server. Please not that I always refer to client - server communication. What Atlas does internally so ensure high availability doesn’t concern me at all.

Anyway, it is as it is and I can’t change it. That’s unfortunate. I’ll have a look at the heartbeat frequency. Maybe this can be increased and reduces the traffic.

6

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.