AWS Peering with Multi-region Cluster connection problem

Saurabh_Singh8 · October 20, 2023, 8:53am

I have a multi region mongo atlas cluster of 3 nodes us-east-1(Primary), us-east-2, us-west-2.
2 APIs in AWS in us-east-1 and us-west-2
Setup two Peering Connection for eac VPC in my AWS infra. All necessary steps done.

Problem - I can connect to Mongo from API/Lambda running in us-east-1 but cannot connect from us-west-2

"detail":"{\"ClassName\":\"System.TimeoutException\",\"Message\":\"A timeout occurred after 30000ms selecting a server using CompositeServerSelector{ Selectors = ReadPreferenceServerSelector{ ReadPreference = { Mode : Primary } }, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : \\\"1\\\", ConnectionMode : \\\"ReplicaSet\\\", Type : \\\"ReplicaSet\\\", State : \\\"Connected\\\", Servers : [{ ServerId: \\\"{ ClusterId : 1, EndPoint : \\\"Unspecified/prod-usa-cluster-shard-00-00.aaaaaa.mongodb.net:27017\\\" }\\\", EndPoint: \\\"Unspecified/prod-usa-cluster-shard-00-00.aaaaaa.mongodb.net:27017\\\", ReasonChanged: \\\"Heartbeat\\\", State: \\\"Disconnected\\\", ServerVersion: , TopologyVersion: , Type: \\\"Unknown\\\", HeartbeatException: \\\"MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.\\n —> System.TimeoutException: Timed out connecting to 52.206.88.60:27017. Timeout was 00:00:30.\\n at MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)\\n — End of inner exception stack trace —\\n at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnection(CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Servers.ServerMonitor.Heartbeat(CancellationToken cancellationToken)\\\", LastHeartbeatTimestamp: \\\"2023-10-20T08:38:38.6956720Z\\\", , LastUpdateTimestamp: \\\"2023-10-20T08:38:38.6956723Z\\\" }, { ServerId: \\\"{ ClusterId : 1, EndPoint : \\\"Unspecified/prod-usa-cluster-shard-00-01.aaaaaa.mongodb.net:27017\\\" }\\\", EndPoint: \\\"Unspecified/prod-usa-cluster-shard-00-01.aaaaaa.mongodb.net:27017\\\", ReasonChanged: \\\"Heartbeat\\\", State: \\\"Connected\\\", ServerVersion: 6.0.0, TopologyVersion: { \\\"processId\\\" : ObjectId(\\\"653191f7a07a297924746906\\\"), \\\"counter\\\" : NumberLong(8) }, Type: \\\"ReplicaSetSecondary\\\", Tags: \\\"{ nodeType : ELECTABLE, provider : AWS, region : US_WEST_2, workloadType : OPERATIONAL }\\\", WireVersionRange: \\\"[0, 17]\\\", LastHeartbeatTimestamp: \\\"2023-10-20T08:38:39.0250305Z\\\", LastUpdateTimestamp: \\\"2023-10-20T08:38:39.0250309Z\\\" }, { ServerId: \\\"{ ClusterId : 1, EndPoint : \\\"Unspecified/prod-usa-cluster-shard-00-02.aaaaaa.mongodb.net:27017\\\" }\\\", EndPoint: \\\"Unspecified/prod-usa-cluster-shard-00-02.aaaaaa.mongodb.net:27017\\\", ReasonChanged: \\\"Heartbeat\\\", State: \\\"Disconnected\\\", ServerVersion: , TopologyVersion: , Type: \\\"Unknown\\\", HeartbeatException: \\\"MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.\\n —> System.TimeoutException: Timed out connecting to 13.56.111.96:27017. Timeout was 00:00:30.\\n at MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)\\n — End of inner exception stack trace —\\n at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnection(CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Servers.ServerMonitor.Heartbeat(CancellationToken cancellationToken)\\\", LastHeartbeatTimestamp: \\\"2023-10-20T08:38:38.6758014Z\\\", LastUpdateTimestamp: \\\"2023-10-20T08:38:38.6758018Z\\\" }] }.\",\"Data\":null,\"InnerException\":null,\"HelpURL\":null,\"StackTraceString\":\" at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)\\n at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)\\n at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedAsync(IServerSelector selector, ClusterDescription description, Task descriptionChangedTask, TimeSpan timeout, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Clusters.Cluster.SelectServerAsync(IServerSelector selector, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Clusters.IClusterExtensions.SelectServerAndPinIfNeededAsync(ICluster cluster, ICoreSessionHandle session, IServerSelector selector, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Bindings.ReadPreferenceBinding.GetReadChannelSourceAsync(CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Operations.RetryableReadContext.InitializeAsync(CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Operations.RetryableReadContext.CreateAsync(IReadBinding binding, Boolean retryRequested, CancellationToken cancellationToken)\\n at MongoDB.Driver.Core.Operations.FindOperation1.ExecuteAsync(IReadBinding binding, CancellationToken cancellationToken)\\\\n at MongoDB.Driver.OperationExecutor.ExecuteReadOperationAsync[TResult](IReadBinding binding, IReadOperation1 operation, CancellationToken cancellationToken)\\n at MongoDB.Driver.MongoCollectionImpl1.ExecuteReadOperationAsync[TResult](IClientSessionHandle session, IReadOperation1 operation, ReadPreference readPreference, CancellationToken cancellationToken)\\n at MongoDB.Driver.MongoCollectionImpl1.UsingImplicitSessionAsync[TResult](Func2 funcAsync, CancellationToken cancellationToken)\\n at MongoDB.Driver.IAsyncCursorSourceExtensions.ToListAsync[TDocument](IAsyncCursorSource`1 source, CancellationToken cancellationToken)\\n at

Jason_Tran · October 31, 2023, 10:57pm

Hi @Saurabh_Singh8 - Welcome to the community.

As per the Deployments in multiple regions section of the Network Peering documentation:

Atlas deployments in multiple regions must have a peering connection for each Atlas region.
For example: If you have a VPC in Sydney and Atlas deployments in Sydney and Singapore, create two peering connections.

To clarify, when you state you set up two peering connections for each VPC from your AWS infra, does that mean you set up a total of 4 VPC peering connections?

Are you able to try performing a ping test from each of your application’s VPCs to each node in Atlas to see the results. Are you being returned the an IP that exists within the configured Atlas VPC CIDR for each ping test?

Regards,
Jason