Aiming for high performance on single server

Ops and Admin
, ,
1

Hi everyone!

I am currently working on a MongoDB test system on a server for a proof of concept project, to see the limits for a later telecommunication project, but I have some performance issues with the MongoDB on the server.

The specs of the server:
CPU: 2x AMD epyc 7453
RAM: 256GB
Storage: 12x 20TB ultrastar HDD, 2x 1TB NVMe SSD
OS: Debian 11

I’m working with a more or less realistic dummy data, which includes about 8.5kB of data per document (timestamp, IPv6 addresses, random 32 and 64 bit values…etc), in database sizes on the scale of 10…100GB (later scaling up to the terrabyte territory). All test ran on the same server on the localhost address.

Without sharding, replica sets, and with one mongod process, the results were something like this:

  • without indexing, about 60-85.000 inserts per second, depending on how aggregated the data was (eg. inserting 100…1.000…100.000 documents at once)
  • with indexing, depending on the number of indexed fields (and the aggregation), about 30-60.000 inserts per second

This seemed like a realistic range, based on articles and previous tests (also to be clear, these are the speed of the inserts themselves, no data handling is calculated in this). Though the server was not running at full power, it should be capable of higher performance overall.

At this point it seemed to be a good opportunity to test sharding on a single server - the concept was that if more mongod instances ran on the same server, the overall performace would be higher. I used the timestamp as a shard key, in ranged mode, since it’s more or less an “incemental” value and therefore doesn’t really need hashing for appropriate load balancing. And…this is where I lost track.

When I used 3 different shard servers on one SSD, the insertion speed was around 15-40.000 inserts per second. With 11 different shard servers still on one SSD, I got around 15-35.000 inserts per second, and when I switched to 11 different shard servers, each on a dedicated HDD, I got around 1000-30.000 inserts per second, which is very far behind the un-sharded test results.

The CPU was not running on full power, the full system memory is about 25 times more than the size of the database(s), and even a single HDD (or one SSD) should be able to write more data than that (I mean in data speed). Maybe I should test other shard key strategies, change the test scenario in case the storage cache is corrupting the results, but I’m not really sure about that.

Has anyone any suggestions on the topic?

2

Running more than a single instance of mongod/mongod on the same hardware is detrimental to performance. Especially if your client application, load simulato, is also running on the same machine. Yes, you illiminate network latency but you increase resources contention. You cannot do much with 2 cpu without context switching. Shards add a lot of overhead, the only way to have better overall performances is with multiple physical machine.

2 Likes
3

Besides running on one server there are also some other concerns with this deployment.

  1. Using the date field (which is monotomically increaseing) is not the best for a shard key. See here. This can lead to a “hot” shard because the data isn’t actually balanced as time goes on, the older date range will not be used as much as time moves forward. This can cause a small amount of the shards to handle most of the queries thus making your sharding kind of useless.
  2. You have 11 different shards but does your query use the shard key as a prefix? If not you will get scatter gather queries across all 11 shards. Instead of direct and specific queries going to the correct shard only.
  3. Standalone shards are not recommended because if one of the nodes goes down that portion of the data isn’t available since it’s distributed across the shards.
  4. Finally Sharding isn’t a “magic” solution that give instant performance as said before your queries have to be optimized so they are routed by the mongos correctly.
3 Likes
4

Thank you both for your answers.

Yes, after a bit of researching I see why this shard key strategy was not the best idea.

The 11 as a number of shard servers was just an arbitary number came to my mind, it was a test scenario to see how it changes overall performance. Well, after thinking about it (and looking at the databases in each shard server) I see why it didn’t permform as I was hoping for - all of them are just a member of one replica set, and without any other setting, all of them just store the same data, therefore slowing down operation. Though, one thing I noticed: the overall insertsion speed dropped by 10-40%, but the computer itself was writing to 11 different places → therefore all in all it was still writing more data in the same given time, than without sharding.

Just out of curiosity I made a test with zones: I made 5 zones with 5 shards (each with a replica set consisting of one member, yes I know, not very realistic to be honest) with a shard key that should have made more or less a decent balance. Well, the shards store almost the same data size -which were about the fifth of the whole database in size, so to this point it was what I expected. But… it was writing especially slowly. I know using zones in sharding is very much not made for this purpose, but still I was surprised that it was this slower.

For the time we have a few more of these servers at our hands I will make some test with those in a LAN just to see how it performs, but still I’m somehow “disappointed” in the results. With a much older and weaker server I could get about 30-40.000 inserts per second on a similar scenario (I mean without sharding, but the same random data) and I was hoping somehow this could pull the trick, and not just by a factor of 2-3 times. Of course I know multithreading has it’s limitations and sure, no program will probably use all cores and threads to push the hardware to it’s full limits.

5

If the above is really what you observed (all shards holding the same data) then you did not configured sharded clusted. Each shard is a different replica set and each is supposed to hold a different set of data.

I strongly recommend that you take M103 from university.mongodb.com as you seem to lack some fundamental knowledge about sharding vs replica set to do what you are doing.