I’m currently running the following setup and am baffled with what is wrong.
3 x r5.2xlarge EC2 instances (8 CPU / 64GB Memory)
Each server has the following 3 drives setup.
- Data - 150GB - EBS GP3 w/ 3000 IOPS (125MB/s)
- Journal - 10GB - EBS GP3 w/ 3000 IOPS (125MB/s)
- Logs - 10GB - EBS GP3 w/ 3000 IOPS (125MB/s)
I’m running into an issue when I start approaching like 1k Inserts per second. I see the following issues happen during my load testing.
- Write Tickets plummet to 7 of 128
- Disk Util % only on Journal drive is pegged at like 90-100%
- Operation Execution time on Writes jump to like 150-200ms
- IOWAIT is like 5-6%
- Normalized CPU is 60-70% max
Now my first instinct is the drive is maxed out, but what baffles me is that my IOPS are only at 1200/s and I have up to 3000/s. I’ve also tried using provisioned IOPS and it didn’t change anything.
What am I missing here?