Is there a bandwidth/speed limitation with MongoDB? Ideally the insert speeds should come near the file transfer speeds, but it’s 1/3 at best.
Details below.
Server : MongoDB v4.4 community edition, running on Windows 10 Virtual Machine
Client : .NET app using v2.11 .NET Mongo Driver
The application has to save 1.22MB of data every 100 milliseconds, more than that causes latencies in my system which is very serious. Without using compression it takes ~140ms, with compression it takes ~80ms but the system runs many services so there could be delays, so even the ~80ms is not enough. With compression the transfer speed is ~15MBps at best, but the maximal transfer speed I achieved (using a file transfer to the mongo server computer) gave 50MBps.
Moreover, I checked the bandwidth using simple client-server python scripts (on the same Mongo service port), I received a 30MBps transfer speed for transferring 2MB (Which is more than the data I’m required to insert uncompressed) in a while loop . Each send took ~2ms.
Ideally MongoDB would perform as much as the hardware allows, as well as any database product in the market. It’ll be a strange database product indeed that deliberately creates artificial limits to performance
However I’d like to clarify some things:
MongoDB v4.4 community edition, running on Windows 10 Virtual Machine
Any reason why it needs to run inside a virtual machine? In light of the timing requirements, have you tried giving the virtual machine more resources and see if it improves the situation?
the system runs many services so there could be delays
Is the MongoDB server sharing the hardware with other resource-intensive apps?
Without using compression it takes ~140ms, with compression it takes ~80ms
How are you measuring these times? Is it from the app side, or the server side, or others? I’m thinking that if it’s from the app side, then the time would be app+database. Would it be possible to isolate the timings from only the app side and only the database side so we know which one needs improvements?
Also what is the topology: are you running MongoDB as a standalone, or a replica set, or a sharded cluster?
In general, I would recommend you setup MongoDB according to the production notes and operations checklist for best results. Please ensure that the MongoDB server is running by following those recommendations so the server is set up for success and not inadvertently given a suboptimal setting that will hinder it.
Any reason why it needs to run inside a virtual machine? In light of the timing requirements, have you tried giving the virtual machine more resources and see if it improves the situation?
It runs on a cloud service at work similar to Azure or AWS, so the resources are not the issue. Moreover, performance on said VM is better than on another VM with better resources (but different disk configuration).
Is the MongoDB server sharing the hardware with other resource-intensive apps?
No.
How are you measuring these times? Is it from the app side, or the server side, or others? I’m thinking that if it’s from the app side, then the time would be app+database. Would it be possible to isolate the timings from only the app side and only the database side so we know which one needs improvements?
I did the following test: I wrote 60MB using the driver, it took ~10s. I checked mongod logs and noticed that the insert (slow query) took ~400ms. I also checked the network I/O for mongod using the Performance tool and saw that during the inserts the speed was ~70KBps which is weird since my speed calculation gives a better result.
Also what is the topology: are you running MongoDB as a standalone, or a replica set, or a sharded cluster?
Standalone.
In general, I would recommend you setup MongoDB according to the production notes and operations checklist for best results. Please ensure that the MongoDB server is running by following those recommendations so the server is set up for success and not inadvertently given a suboptimal setting that will hinder it.
I have already checked these and the hardware configuration is adequate.
Thanks for the details. A couple of pointers that may be helpful:
I checked mongod logs and noticed that the insert (slow query) took ~400ms.
Ideally we should not see any slow queries in the logs. In my experience, the most typical reason for a database’s slow operation are:
The operation is actually slow (e.g. unindexed queries)
The operation is blocked waiting for another slow operations (e.g. if you have a lot of slow queries some of your queries may just wait to get their turn to do their job)
Need more hardware (e.g. the disk is not fast enough to serve your workload)
Usually I would suggest to determine the main issue in that order. That is, if everything else is tuned to perfection, then maybe the hardware need improvements.
during the inserts the speed was ~70KBps which is weird since my speed calculation gives a better result.
If I understand correctly, the Python script was not writing to disk (which the database server must do), so perhaps that’s one avenue for investigation.
Maybe the output of mongostat could be helpful here: it provides a high level overview of how stressed the server is.
Standalone.
For production deployment, I would encourage you to use a replica set. A standalone should only be used for development purposes.
Also you might be interested in checking out the free MongoDB University course M201 MongoDB Performance which touches on different performance considerations.
