Disk Utilization pegged at near 100%

I’m currently running the following setup and am baffled with what is wrong.

3 x r5.2xlarge EC2 instances (8 CPU / 64GB Memory)

Each server has the following 3 drives setup.

  • Data - 150GB - EBS GP3 w/ 3000 IOPS (125MB/s)
  • Journal - 10GB - EBS GP3 w/ 3000 IOPS (125MB/s)
  • Logs - 10GB - EBS GP3 w/ 3000 IOPS (125MB/s)

I’m running into an issue when I start approaching like 1k Inserts per second. I see the following issues happen during my load testing.

  • Write Tickets plummet to 7 of 128
  • Disk Util % only on Journal drive is pegged at like 90-100%
  • Operation Execution time on Writes jump to like 150-200ms
  • IOWAIT is like 5-6%
  • Normalized CPU is 60-70% max

Now my first instinct is the drive is maxed out, but what baffles me is that my IOPS are only at 1200/s and I have up to 3000/s. I’ve also tried using provisioned IOPS and it didn’t change anything.

What am I missing here?