High Disk I/O% on secondary node

We have a cluster with five nodes, three in AWS us-east-1 and two in us-west-1. Our cluster was restarted on 9/30 and since then, both of the secondary nodes in us-west-1 have been relentlessly hammering us with High I/O% on data partition alerts. Looking at monitoring, our other secondary nodes, in us-east-1, are humming along without issue, but the us-west-1 nodes are constantly at or above 90% util.

What can we do to address this? I thought of removing the us-west-1 nodes and re-adding them, but wanted to ask here before I take that step. Thanks in advance.

It sounds to me as though unindexed queries are running against those secondaries that you may not be aware of. Have you explored the Atlas real-time performance panel for those particular nodes to see if you can see any active queries there? Another option would be to explore the logs. I recommend opening up a support case for mongodb to help you diagnose what’s going on

1 Like