Changing the Cluster Tier based Kafka/Amazon SQS lag

MongoDB Atlas
1

I would like your opinion based on the following scenario.

There are 2 microservices, one that consumes data from Kafka and another that consumes data from Amazon SQS. Both microservices persist data in a MongoDB database. The image below demonstrates the scenario better.

cluster-tier-changing
cluster-tier-changing1105×663 29.5 KB

At specific times of the day, these microservices receive a very large volume of events from Kafka and Amazon SQS. The cluster Tier is M30, however at these times M30 is not sufficient and it is necessary to escalate to M50. The problem is that the current trigger used by MongoDB based on memory or CPU waits 1 hour to automatically scale, which makes the operation unfeasible.

We are evaluating possible approaches to scale the cluster tier from M30 to M50, using the number of pending events (lag) in Kafka and Amazon SQS as a trigger. Does anyone have a suggestion?

2

Hi @Augusto_Colomb ,

I recommend using the Atlas in-app support to connect with MongoDB Support to see if they can assist you specifically with Atlas thresholds for tier scaling.

To drive the tier scaling from an external metric some tooling of your own combined with Programatic Access(api, terraform,cdk) or Atlas CLI may work for you.

3

Hi @chris,
Thanks for your response and recommendations. We are tending to use requests for MongoDB via API, we initially evaluated keda (Apache Kafka | KEDA).

We are evaluating other alternatives that can act as intermediaries listening to Kafka/Amazon SQS and making requests to MongoDB.