I have been designing a load testing suite for my rest APIs that use Mongo behind the scenes. The idea is to benchmark API calls (including persistence to Mongo ). The suite is a highly concurrent set of invocations and I need a couple of suggestions
Mongo caches plan - therefore, after the first run, my average response time drops leading to a false good number. I understand that I could clear the plan cache - but
I may not have the required levels of access to clear the plan cache
I will have to run this after every invocation of my test endpoint and given concurrent invocations of the subject, it may not be as effective as invocations may have happened concurrently
Idempotency
Some of my operations are not idempotent - is there a way I could pass in a flag to not truly commit to persistence layer or rollback at the end of the operation so as to not alter the state?
This is to avoid building additional layering to reset state after every test, which by the way may not be possible effectively given concurrent invocations.
Can someone let me know how Mongo itself runs benchmarking and if we could use some of those strategies here? Or if there is a better way to do what I’m trying to do ?
Why would you want to test the performance of a cold system? That is counter intuitive because you want to know how your system behaves under normal circumstances. Being cold with caches empty and working set on disks out of RAM is not normal circumstances and your system is not supposed to be stopped or restarted often.
Unless, stopping and restarting is part of your normal use cases. But in this case, you really have to stop and start every time.
About 2.
If you do not truly commit to the persistence layer in order to avoid resetting the state then your are not really testing the performance of your end-to-end system.
The reason behind using a cold system for benchmarking was to benchmark with the worstcase scenario as opposed to the best case. While the system is warm, the perf tests will make the test system warmer than what it might normally be under normal circumstances - and so was leaning towards taking a worstcase benchmark. However, I have come around a little bit after posting this : I would stick to the “warmed” approach - but I look at the “max response” numbers separately alongside average and p99s.
Agree with your point on rescinding the persistence request via a rollback will not lead us to a similar benchmarking.