EventGet 50% off your ticket to MongoDB.local NYC on May 2. Use code Web50!Learn more >>
MongoDB Developer
MongoDB
plus
Sign in to follow topics
MongoDB Developer Centerchevron-right
Developer Topicschevron-right
Productschevron-right
MongoDBchevron-right

MongoDB Network Compression: A Win-Win

Brian Leonard6 min read • Published Nov 03, 2021 • Updated Oct 12, 2022
MongoDB
Facebook Icontwitter iconlinkedin icon
Rate this tutorial
star-empty
star-empty
star-empty
star-empty
star-empty

MongoDB Network Compression: A Win-Win

An under-advertised feature of MongoDB is its ability to compress data between the client and the server. The CRM company Close has a really nice article on how compression reduced their network traffic from about 140 Mbps to 65 Mpbs. As Close notes, with cloud data transfer costs ranging from $0.01 per GB and up, you can get a nice little savings with a simple configuration change.
mongodb-network-compression-chart
MongoDB supports the following compressors:
  • zlib (Available starting in MongoDB 3.6)
  • zstd (Available starting in MongoDB 4.2)
Enabling compression from the client simply involves installing the desired compression library and then passing the compressor as an argument when you connect to MongoDB. For example:
This article provides two tuneable Python scripts, read-from-mongo.py and write-to-mongo.py, that you can use to see the impact of network compression yourself.

Setup

Client Configuration

Edit params.py and at a minimum, set your connection string. Other tunables include the amount of bytes to read and insert (default 10 MB) and the batch size to read (100 records) and insert (1 MB):

Compression Library

The snappy compression in Python requires the python-snappy package.
pip3 install python-snappy
The zstd compression requires the zstandard package
pip3 install zstandard
The zlib compression is native to Python.

Sample Data

My read-from-mongo.py script uses the Sample AirBnB Listings Dataset but ANY dataset will suffice for this test.
The write-to-mongo.py script generates sample data using the Python package Faker.
pip3 install faker

Execution

Read from Mongo

The cloud providers notably charge for data egress, so anything that reduces network traffic out is a win.
Let's first run the script without network compression (the default):
You've obviously noticed the reported Megabytes out (188 MB) are more than 18 times our test size of 10 MBs. There are several reasons for this, including other workloads running on the server, data replication to secondary nodes, and the TCP packet being larger than just the data. Focus on the delta between the other tests runs.
The script accepts an optional compression argument, that must be either snappy, zlib or zstd. Let's run the test again using snappy, which is known to be fast, while sacrificing some compression:
With snappy compression, our reported bytes out were about 62 MBs fewer. That's a 33% savings. But wait, the 10 MBs of data was read in 10 fewer seconds. That's also a 33% performance boost!
Let's try this again using zlib, which can achieve better compression, but at the expense of performance.
zlib compression supports an optional compression level. For this test I've set it to 9 (max compression).
With zlib compression configured at its maximum compression level, we were able to achieve a 64% reduction in network egress, although it took 4 seconds longer. However, that's still a 19% performance improvement over using no compression at all.
Let's run a final test using zstd, which is advertised to bring together the speed of snappy with the compression efficiency of zlib:
And sure enough, zstd lives up to its reputation, achieving 68% percent improvement in compression along with a 55% improvement in performance!

Write to Mongo

The cloud providers often don't charge us for data ingress. However, given the substantial performance improvements with read workloads, what can be expected from write workloads?
The write-to-mongo.py script writes a randomly generated document to the database and collection configured in params.py, the default being test.network_compression_test.
As before, let's run the test without compression:
So it took 15 seconds to write 27,778 records. Let's run the same test with zstd compression:
Our reported megabytes in are reduced by 62%. However, our write performance remained identical. Personally, I think most of this is due to the time it takes the Faker library to generate the sample data. But having gained compression without a performance impact it is still a win.

Measurement

There are a couple of options for measuring network traffic. This script is using the db.serverStatus() physicalBytesOut and physicalBytesIn, reporting on the delta between the reading at the start and end of the test run. As mentioned previously, our measurements are corrupted by other network traffic occuring on the server, but my tests have shown a consistent improvement when run. Visually, my results achieved appear as follows:
megabytes out seconds to read
Another option would be using a network analysis tool like Wireshark. But that's beyond the scope of this article for now.
Bottom line, compression reduces network traffic by more than 60%, which is in line with the improvement seen by Close. More importantly, compression also had a dramatic improvement on read performance. That's a Win-Win.

Facebook Icontwitter iconlinkedin icon
Rate this tutorial
star-empty
star-empty
star-empty
star-empty
star-empty
Related
Article

5 Key Takeaways from Hacktoberfest 2020


Sep 23, 2022 | 8 min read
News & Announcements

MongoDB Academia - Introduction to Modern Databases


Sep 23, 2022 | 4 min read
Tutorial

Local Development with the MongoDB Atlas CLI and Docker


Feb 06, 2023 | 9 min read
Article

TableCheck: Empowering Restaurants with Best-in-Class Booking Tools Powered by MongoDB


Dec 14, 2022 | 4 min read
Table of Contents
  • MongoDB Network Compression: A Win-Win