Prometheus Integration: Support for Cross-Region Atlas Clusters

Ops and Admin
1

Cluster: 5 instances split between 2 regions

Created an integration for prometheus. After applying the config and restarting prometheus I am now seeing metrics flow to prometheus from the cluster. But they are incomplete. Only 3 of the 5 nodes in the cluster are getting scraped. The other 2 are not. The 2 that are not represented are not in the region that is hosting prometheus (the 3 that reporting are). Does the prometheus atlas integration not support cross-region? If not, please document. If it does, please provide information on how it can be achieved.

3

Hi @andrew_morcomb - Welcome to the community :wave:

For the discussion i’ll just reference the following based off your above information but please correct me if this interpretation / example is incorrect:

  1. Region 1 - 3 nodes (prometheus server able to scrape off these 3 nodes)
  2. Region 2 - 2 nodes (prometheus server not able to scrape off these 2 nodes)
  3. Prometheus server exists on region 1 in your cloud provider (assuming AWS but please advise if GCP / Azure).

One thing I can think off of the top of my head is if you have a VPC peering connection configured to/from the AWS VPC where the prometheus server exists to each of the Atlas region VPC’s. That is:

AWS VPC (Prometheus server) Region 1 <-- peering connection 1 --> Atlas VPC Region 1
AWS VPC (Promeheus server) Region 1 <-- peering connection 2 --> Atlas VPC Region 2

Note: I am assuming vpc peering due to the prometheus agent being able to scrape off only a singular region:

Deployments in Multiple Regions
Atlas deployments in multiple regions must have a peering connection for each Atlas region.

If you believe you have this set up with 2 vpc peering connections then you can maybe do some network tests to ensure the client from the region where the prometheus server exists is able to connect to both regions on the Atlas end via the vpc peering connections just as a troubleshooting step.

If this is not the case, you may wish to check with the Atlas in-app chat support team to see if there are any limitations on this set up from the Atlas end. Although private endpoints are currently not supported for the prometheus integration as of today. Please note they won’t have any insight into your AWS networking configurations on your end.

Hope the above hopes in any way.

Regards,
Jason