Client Disconnect on Cloud Run Docker .Net container

Ray_Simpson · July 11, 2022, 4:40pm

Hello,

I have been trying for days to write a simple proof of concept web api that writes 5 entries into a mongodb. The application is written in .net 3.1, it is the default weather sample that gets generated when a new web api is created. I have altered the api to write the weather samples to a collection, it is built inside of a docker container that has been pushed and setup as a service in Google Cloud Run. On my local instance the docker container works perfectly, writes the 5 entries into my local mongodb. However, the cloud run service errors out stating it is getting a ```

2022-07-07 15:24:49.688 MST —> System.IO.EndOfStreamException: Attempted to read past the end of the stream.
Default

Every single time. I was using the free cluster, but was unable to see the mongodb logs so I purchased the paid version. According to the log files, right after authentication, it says “Interrupted operation as its client disconnected”. I am using the latest Mongo Db.driver (2.16.1). For a more detailed description and to see the log files entries please go to https://stackoverflow.com/questions/72905203/cannot-connect-to-mongodb-atlas-from-google-cloud-run-docker-container on Stack overflow. At this point I do not know where to concentrate my efforts, (is it on the Cloud Run side or Mongodb side?), any help would be greatly appreciated.

R

Dmitry_Lukyanov · July 12, 2022, 1:37am

hey, have you configured network access on atlas side? Can you connect to your atlas cluster if you don’t use docker, like with a simple console app?

Ray_Simpson · July 12, 2022, 5:22pm

Dmitry thank you for responding,

I have been able to connect to the atlas connection from my development machine. Try to make sure it could connect, I added 0.0.0.0/16 to the white list to make sure it wasn’t a connection side issue. I am currently trying to get a VM of Mongo Express on the cloud side to work, to see if I can connect to atlas from there. If I am able to do so, then I would assume that its an issue with the driver for .Net when running in a container in GCP as it works just fine outside of the cloud.

R

Ray_Simpson · July 14, 2022, 5:04pm

I think I am going to give up on this and ask for a refund for my Mongo DB Atlas service. I have been trying non-stop to get this to work for a solid week and I just can’t try to make the round peg fit in the square hole anymore. Maybe my approach is off, but I thought making a docker container that was run on the google cloud and connected to Atlas would be a breeze, but it turns out to be a hurricane instead. To those that responded to my post, I appreciate it.

James_Kovacs · July 14, 2022, 6:20pm

Hi, Ray,

We are sorry to hear about your difficulties in connecting from Docker on GCP to MongoDB Atlas. It is interesting that Docker locally is able to connect without problem. I’m not sure what peculiarities of Docker running on GCP would cause the problems that you’re encountering. It would be helpful to have the full stack trace from the logs that you shared initially. This will help us determine whether the connection failure is when attempting to reach DNS, establishing monitoring / RTT connections, or set up the connection pool connections. Thanks in advance for any additional information that you can provide.

Sincerely,
James

Ray_Simpson · July 14, 2022, 6:58pm

Hey James,
Here it is:

{“t”:{"$date":“2022-07-10T06:15:59.771+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“127.0.0.1:54648”,“uuid”:“cdeb7bd0-e4f9-4725-8794-3999d1342e12”,“connectionId”:783,“connectionCount”:36}}
{“t”:{"$date":“2022-07-10T06:15:59.771+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“127.0.0.1:54650”,“uuid”:“483d20d5-7ceb-49aa-860e-75e2631d0c43”,“connectionId”:784,“connectionCount”:37}}
{“t”:{"$date":“2022-07-10T06:15:59.782+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“127.0.0.1:54652”,“uuid”:“42b8bcb1-9888-4ae5-8ea5-fc5edf120e47”,“connectionId”:785,“connectionCount”:38}}
{“t”:{"$date":“2022-07-10T06:15:59.783+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn784”,“msg”:“Connection ended”,“attr”:{“remote”:“127.0.0.1:54650”,“uuid”:“483d20d5-7ceb-49aa-860e-75e2631d0c43”,“connectionId”:784,“connectionCount”:37}}
{“t”:{"$date":“2022-07-10T06:15:59.783+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn783”,“msg”:“Connection ended”,“attr”:{“remote”:“127.0.0.1:54648”,“uuid”:“cdeb7bd0-e4f9-4725-8794-3999d1342e12”,“connectionId”:783,“connectionCount”:36}}
{“t”:{"$date":“2022-07-10T06:15:59.783+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“127.0.0.1:54654”,“uuid”:“d48b39a5-531e-430b-9a63-445d49aa0521”,“connectionId”:786,“connectionCount”:37}}
{“t”:{"$date":“2022-07-10T06:15:59.805+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn785”,“msg”:“client metadata”,“attr”:{“remote”:“127.0.0.1:54652”,“client”:“conn785”,“doc”:{“driver”:{“name”:“mongo-go-driver”,“version”:“v1.7.2+prerelease”},“os”:{“type”:“linux”,“architecture”:“amd64”},“platform”:“go1.17.10”,“application”:{“name”:“MongoDB Automation Agent v12.0.6.7562 (git: cddb628636c9576fed56ce13bdc0a3d24e65ca1c)”}}}}
{“t”:{"$date":“2022-07-10T06:15:59.806+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“127.0.0.1:54656”,“uuid”:“7c1061f4-0575-4b37-834d-5ea0c2640c9c”,“connectionId”:787,“connectionCount”:38}}
{“t”:{"$date":“2022-07-10T06:15:59.806+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn786”,“msg”:“client metadata”,“attr”:{“remote”:“127.0.0.1:54654”,“client”:“conn786”,“doc”:{“driver”:{“name”:“mongo-go-driver”,“version”:“v1.7.2+prerelease”},“os”:{“type”:“linux”,“architecture”:“amd64”},“platform”:“go1.17.10”,“application”:{“name”:“MongoDB Automation Agent v12.0.6.7562 (git: cddb628636c9576fed56ce13bdc0a3d24e65ca1c)”}}}}
{“t”:{"$date":“2022-07-10T06:15:59.818+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn787”,“msg”:“client metadata”,“attr”:{“remote”:“127.0.0.1:54656”,“client”:“conn787”,“doc”:{“driver”:{“name”:“mongo-go-driver”,“version”:“v1.7.2+prerelease”},“os”:{“type”:“linux”,“architecture”:“amd64”},“platform”:“go1.17.10”,“application”:{“name”:“MongoDB Automation Agent v12.0.6.7562 (git: cddb628636c9576fed56ce13bdc0a3d24e65ca1c)”}}}}
{“t”:{"$date":“2022-07-10T06:15:59.818+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20249, “ctx”:“conn787”,“msg”:“Authentication failed”,“attr”:{“mechanism”:“SCRAM-SHA-256”,“speculative”:true,“principalName”:“mms-automation”,“authenticationDatabase”:“admin”,“remote”:“127.0.0.1:54656”,“extraInfo”:{},“error”:“BadValue: SCRAM-SHA-256 authentication is disabled”}}
{“t”:{"$date":“2022-07-10T06:15:59.824+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn787”,“msg”:“Authentication succeeded”,“attr”:{“mechanism”:“SCRAM-SHA-1”,“speculative”:false,“principalName”:“mms-automation”,“authenticationDatabase”:“admin”,“remote”:“127.0.0.1:54656”,“extraInfo”:{}}}
{“t”:{"$date":“2022-07-10T06:16:10.382+00:00”},“s”:“I”, “c”:“SHARDING”, “id”:20997, “ctx”:“conn782”,“msg”:“Refreshed RWC defaults”,“attr”:{“newDefaults”:{}}}
{“t”:{"$date":“2022-07-10T06:16:10.386+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“192.168.248.2:44416”,“uuid”:“f97bd60d-3747-4864-8b45-58770eda6315”,“connectionId”:788,“connectionCount”:39}}
{“t”:{"$date":“2022-07-10T06:16:10.386+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“192.168.248.2:44418”,“uuid”:“439e6612-715e-4290-bfcb-afa40f51fcc2”,“connectionId”:789,“connectionCount”:40}}
{“t”:{"$date":“2022-07-10T06:16:10.408+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn789”,“msg”:“client metadata”,“attr”:{“remote”:“192.168.248.2:44418”,“client”:“conn789”,“doc”:{“driver”:{“name”:“mongo-go-driver”,“version”:“v1.7.2+prerelease”},“os”:{“type”:“linux”,“architecture”:“amd64”},“platform”:“go1.17.10”,“application”:{“name”:“MongoDB Automation Agent v12.0.6.7562 (git: cddb628636c9576fed56ce13bdc0a3d24e65ca1c)”}}}}
{“t”:{"$date":“2022-07-10T06:16:10.408+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn788”,“msg”:“client metadata”,“attr”:{“remote”:“192.168.248.2:44416”,“client”:“conn788”,“doc”:{“driver”:{“name”:“mongo-go-driver”,“version”:“v1.7.2+prerelease”},“os”:{“type”:“linux”,“architecture”:“amd64”},“platform”:“go1.17.10”,“application”:{“name”:“MongoDB Automation Agent v12.0.6.7562 (git: cddb628636c9576fed56ce13bdc0a3d24e65ca1c)”}}}}
{“t”:{"$date":“2022-07-10T06:16:10.409+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“192.168.248.2:44420”,“uuid”:“df1fd54b-e6fd-48b3-afc4-2afd2f5bc63a”,“connectionId”:790,“connectionCount”:41}}
{“t”:{"$date":“2022-07-10T06:16:10.419+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn790”,“msg”:“client metadata”,“attr”:{“remote”:“192.168.248.2:44420”,“client”:“conn790”,“doc”:{“driver”:{“name”:“mongo-go-driver”,“version”:“v1.7.2+prerelease”},“os”:{“type”:“linux”,“architecture”:“amd64”},“platform”:“go1.17.10”,“application”:{“name”:“MongoDB Automation Agent v12.0.6.7562 (git: cddb628636c9576fed56ce13bdc0a3d24e65ca1c)”}}}}
{“t”:{"$date":“2022-07-10T06:16:10.430+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn790”,“msg”:“Authentication succeeded”,“attr”:{“mechanism”:“SCRAM-SHA-256”,“speculative”:true,“principalName”:"__system",“authenticationDatabase”:“local”,“remote”:“192.168.248.2:44420”,“extraInfo”:{}}}
{“t”:{"$date":“2022-07-10T06:16:10.431+00:00”},“s”:“I”, “c”:"-", “id”:20883, “ctx”:“conn788”,“msg”:“Interrupted operation as its client disconnected”,“attr”:{“opId”:15903}}
{“t”:{"$date":“2022-07-10T06:16:10.432+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn789”,“msg”:“Connection ended”,“attr”:{“remote”:“192.168.248.2:44418”,“uuid”:“439e6612-715e-4290-bfcb-afa40f51fcc2”,“connectionId”:789,“connectionCount”:40}}
{“t”:{"$date":“2022-07-10T06:16:10.433+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn788”,“msg”:“Connection ended”,“attr”:{“remote”:“192.168.248.2:44416”,“uuid”:“f97bd60d-3747-4864-8b45-58770eda6315”,“connectionId”:788,“connectionCount”:39}}
{“t”:{"$date":“2022-07-10T06:16:10.434+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn790”,“msg”:“Connection ended”,“attr”:{“remote”:“192.168.248.2:44420”,“uuid”:“df1fd54b-e6fd-48b3-afc4-2afd2f5bc63a”,“connectionId”:790,“connectionCount”:38}}
{“t”:{"$date":“2022-07-10T06:16:15.912+00:00”},“s”:“I”, “c”:“STORAGE”, “id”:22430, “ctx”:“Checkpointer”,“msg”:“WiredTiger message”,“attr”:{“message”:"[1657433775:912061][4435:0x7f96a8fff700], WT_SESSION.checkpoint: [WT_VERB_CHECKPOINT_PROGRESS] saving checkpoint snapshot min: 423, snapshot max: 423 snapshot count: 0, oldest timestamp: (1657433535, 1) , meta checkpoint timestamp: (1657433767, 1) base write gen: 1"}}
{“t”:{"$date":“2022-07-10T06:16:27.462+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“192.168.248.4:50328”,“uuid”:“c8c3b09e-ddee-4aa4-a63e-34f1051d2627”,“connectionId”:791,“connectionCount”:39}}
{“t”:{"$date":“2022-07-10T06:16:27.473+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn791”,“msg”:“client metadata”,“attr”:{“remote”:“192.168.248.4:50328”,“client”:“conn791”,“doc”:{“driver”:{“name”:“NetworkInterfaceTL”,“version”:“5.0.9”},“os”:{“type”:“Linux”,“name”:“CentOS Linux release 7.9.2009 (Core)”,“architecture”:“x86_64”,“version”:“Kernel 3.10.0-1160.66.1.el7.x86_64”}}}}
{“t”:{"$date":“2022-07-10T06:16:27.485+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn791”,“msg”:“Authentication succeeded”,“attr”:{“mechanism”:“SCRAM-SHA-256”,“speculative”:true,“principalName”:"__system",“authenticationDatabase”:“local”,“remote”:“192.168.248.4:50328”,“extraInfo”:{}}}
{“t”:{"$date":“2022-07-10T06:16:27.486+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“192.168.248.4:50334”,“uuid”:“db907fee-ff31-431d-9179-ef32db6cfb46”,“connectionId”:792,“connectionCount”:40}}
{“t”:{"$date":“2022-07-10T06:16:27.594+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn792”,“msg”:“client metadata”,“attr”:{“remote”:“192.168.248.4:50334”,“client”:“conn792”,“doc”:{“driver”:{“name”:“NetworkInterfaceTL”,“version”:“5.0.9”},“os”:{“type”:“Linux”,“name”:“CentOS Linux release 7.9.2009 (Core)”,“architecture”:“x86_64”,“version”:“Kernel 3.10.0-1160.66.1.el7.x86_64”}}}}
{“t”:{"$date":“2022-07-10T06:16:27.601+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn792”,“msg”:“Authentication succeeded”,“attr”:{“mechanism”:“SCRAM-SHA-256”,“speculative”:true,“principalName”:"__system",“authenticationDatabase”:“local”,“remote”:“192.168.248.4:50334”,“extraInfo”:{}}}
{“t”:{"$date":“2022-07-10T06:16:33.487+00:00”},“s”:“I”, “c”:“SHARDING”, “id”:20997, “ctx”:“conn782”,“msg”:“Refreshed RWC defaults”,“attr”:{“newDefaults”:{}}}
{“t”:{"$date":“2022-07-10T06:16:33.490+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“192.168.248.2:44434”,“uuid”:“8ba9aa9f-9d58-42dd-9f8e-9e1de6fa241f”,“connectionId”:793,“connectionCount”:41}}
{“t”:{"$date":“2022-07-10T06:16:33.491+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“192.168.248.2:44436”,“uuid”:“6b1f202b-7db4-45ba-80ce-6302593880bc”,“connectionId”:794,“connectionCount”:42}}
{“t”:{"$date":“2022-07-10T06:16:33.511+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn794”,“msg”:“client metadata”,“attr”:{“remote”:“192.168.248.2:44436”,“client”:“conn794”,“doc”:{“driver”:{“name”:“mongo-go-driver”,“version”:“v1.7.2+prerelease”},“os”:{“type”:“linux”,“architecture”:“amd64”},“platform”:“go1.17.10”,“application”:{“name”:“MongoDB Automation Agent v12.0.6.7562 (git: cddb628636c9576fed56ce13bdc0a3d24e65ca1c)”}}}}
{“t”:{"$date":“2022-07-10T06:16:33.511+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn793”,“msg”:“client metadata”,“attr”:{“remote”:“192.168.248.2:44434”,“client”:“conn793”,“doc”:{“driver”:{“name”:“mongo-go-driver”,“version”:“v1.7.2+prerelease”},“os”:{“type”:“linux”,“architecture”:“amd64”},“platform”:“go1.17.10”,“application”:{“name”:“MongoDB Automation Agent v12.0.6.7562 (git: cddb628636c9576fed56ce13bdc0a3d24e65ca1c)”}}}}
{“t”:{"$date":“2022-07-10T06:16:33.512+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22943, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“192.168.248.2:44438”,“uuid”:“22f52e37-a1ee-4cf4-911b-1347710f8697”,“connectionId”:795,“connectionCount”:43}}
{“t”:{"$date":“2022-07-10T06:16:33.522+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:51800, “ctx”:“conn795”,“msg”:“client metadata”,“attr”:{“remote”:“192.168.248.2:44438”,“client”:“conn795”,“doc”:{“driver”:{“name”:“mongo-go-driver”,“version”:“v1.7.2+prerelease”},“os”:{“type”:“linux”,“architecture”:“amd64”},“platform”:“go1.17.10”,“application”:{“name”:“MongoDB Automation Agent v12.0.6.7562 (git: cddb628636c9576fed56ce13bdc0a3d24e65ca1c)”}}}}
{“t”:{"$date":“2022-07-10T06:16:33.533+00:00”},“s”:“I”, “c”:“ACCESS”, “id”:20250, “ctx”:“conn795”,“msg”:“Authentication succeeded”,“attr”:{“mechanism”:“SCRAM-SHA-256”,“speculative”:true,“principalName”:"__system",“authenticationDatabase”:“local”,“remote”:“192.168.248.2:44438”,“extraInfo”:{}}}
{“t”:{"$date":“2022-07-10T06:16:33.535+00:00”},“s”:“I”, “c”:"-", “id”:20883, “ctx”:“conn793”,“msg”:“Interrupted operation as its client disconnected”,“attr”:{“opId”:16607}}
{“t”:{"$date":“2022-07-10T06:16:33.536+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn794”,“msg”:“Connection ended”,“attr”:{“remote”:“192.168.248.2:44436”,“uuid”:“6b1f202b-7db4-45ba-80ce-6302593880bc”,“connectionId”:794,“connectionCount”:42}}
{“t”:{"$date":“2022-07-10T06:16:33.536+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn793”,“msg”:“Connection ended”,“attr”:{“remote”:“192.168.248.2:44434”,“uuid”:“8ba9aa9f-9d58-42dd-9f8e-9e1de6fa241f”,“connectionId”:793,“connectionCount”:41}}

James_Kovacs · July 14, 2022, 7:29pm

Hi, Ray,

Thank you for providing the mongod logs. From these I can see that the .NET/C# Driver isn’t even attempting to connect. These would show up in the client metadata with a driver name of mongo-csharp-driver whereas all we see are the mongo-go-driver and intracluster connections. (The mongo-go-driver is used by Atlas for various monitoring and orchestration functions.)

It would be helpful to have the full logs including stack traces from your client application. That will hopefully allow us to see where the failure is happening that is preventing your connection to your MongoDB Atlas cluster.

Thank you for providing the data to help us look into this issue with you.

Sincerely,
James

Ray_Simpson · July 14, 2022, 8:56pm

Blockquote
A timeout occurred after 120000ms selecting a server using CompositeServerSelector{ Selectors = MongoDB.Driver.MongoClient+AreSessionsSupportedServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 }, OperationsCountServerSelector }. Client view of cluster state is { ClusterId : “1”, ConnectionMode : “ReplicaSet”, Type : “ReplicaSet”, State : “Disconnected”, Servers : [{ ServerId: “{ ClusterId : 1, EndPoint : “Unspecified/ac-xxxxxxx-shard-00-00-pri.xxxxxx.mongodb.net:27017” }”, EndPoint: “Unspecified/ac-xxxxxxx-shard-00-00-pri.xxxxxx.mongodb.net:27017”, ReasonChanged: “Heartbeat”, State: “Disconnected”, ServerVersion: , TopologyVersion: , Type: “Unknown”, HeartbeatException: “MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
—> System.TimeoutException: Timed out connecting to 192.168.248.2:27017. Timeout was 00:02:00.
at MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
— End of inner exception stack trace —
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnection(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.Heartbeat(CancellationToken cancellationToken)”, LastHeartbeatTimestamp: “2022-07-14T20:42:18.6034577Z”, LastUpdateTimestamp: “2022-07-14T20:42:18.6034634Z” }, { ServerId: “{ ClusterId : 1, EndPoint : “Unspecified/ac-xxxxxxx-shard-00-01-pri.xxxxxx.mongodb.net:27017” }”, EndPoint: “Unspecified/ac-xxxxxxx-shard-00-01-pri.xxxxxx.mongodb.net:27017”, ReasonChanged: “Heartbeat”, State: “Disconnected”, ServerVersion: , TopologyVersion: , Type: “Unknown”, HeartbeatException: “MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
—> System.TimeoutException: Timed out connecting to 192.168.248.3:27017. Timeout was 00:02:00.
at MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
— End of inner exception stack trace —
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnection(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.Heartbeat(CancellationToken cancellationToken)”, LastHeartbeatTimestamp: “2022-07-14T20:42:18.6013861Z”, LastUpdateTimestamp: “2022-07-14T20:42:18.6013965Z” }, { ServerId: “{ ClusterId : 1, EndPoint : “Unspecified/ac-xxxxxxx-shard-00-02-pri.xxxxxx.mongodb.net:27017” }”, EndPoint: “Unspecified/ac-xxxxxxx-shard-00-02-pri.xxxxxx.mongodb.net:27017”, ReasonChanged: “Heartbeat”, State: “Disconnected”, ServerVersion: , TopologyVersion: , Type: “Unknown”, HeartbeatException: “MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server.
—> System.TimeoutException: Timed out connecting to 192.168.248.4:27017. Timeout was 00:02:00.
at MongoDB.Driver.Core.Connections.TcpStreamFactory.Connect(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.SslStreamFactory.CreateStream(EndPoint endPoint, CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
— End of inner exception stack trace —
at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelper(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Connections.BinaryConnection.Open(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.InitializeConnection(CancellationToken cancellationToken)
at MongoDB.Driver.Core.Servers.ServerMonitor.Heartbeat(CancellationToken cancellationToken)”, LastHeartbeatTimestamp: “2022-07-14T20:42:18.8760889Z”, LastUpdateTimestamp: “2022-07-14T20:42:18.8760893Z” }] }.

Here is the stack from my program. The program itself is very simple:

var connectionString = "mongodb+srv://xxxmongodbuser:Gpr2Un3tlRdcxxxxx@jxxx-cluster-pri.xxxxxx.mongodb.net/?retryWrites=true&w=majority";
            var settings = MongoClientSettings.FromConnectionString(connectionString);
            settings.ConnectTimeout = TimeSpan.FromMinutes(1);
            settings.SocketTimeout = TimeSpan.FromMinutes(1);
            settings.ServerSelectionTimeout = TimeSpan.FromMinutes(1);
            settings.SslSettings = new SslSettings
            {
                EnabledSslProtocols = System.Security.Authentication.SslProtocols.Tls12
            };
            settings.UseTls = true;
            mongoClient = new MongoClient(settings);
}

public static void SaveData(WeatherForecast[] weatherForecasts)
        {
            var database = mongoClient.GetDatabase("WeatherDB");
            var collection = database.GetCollection<WeatherForecast>("Weather");
            collection.InsertMany(weatherForecasts);
        }

James_Kovacs · July 14, 2022, 10:00pm

Hi, Ray,

Thank you for providing the stack trace and program. Both are very helpful in understanding the root cause.

I can see from your connection string and StackOverflow question that you are using VPC peering. The driver has created monitoring connections to the cluster nodes and attempted to heartbeat with each of them. These heartbeats are failing, which explains why the mongod logs show no connections from the driver. The monitoring connections and heartbeats are not even reaching your Atlas cluster.

It would be helpful to determine if the problem is with the VPC peering setup or another problem. I would suggest removing VPC peering from the equation (at least temporarily) and seeing if you can connect from your Docker-hosted application to your MongoDB Atlas cluster.

Ensure your Docker host’s IP address is included in your IP Access List for your Atlas project.
Use the non-VPC connection string to connect to the cluster. Rather than jxxx-cluster-pri.xxxxxx.mongodb.net the FQDN should look similar to jxxx-cluster.xxxxxx.mongodb.net. Note the lack of -pri in the FQDN.

If a non-VPC connection also fails, please provide the application logs so we can determine if the monitoring threads are still unable to successfully connect to the cluster. If you are able to connect, then the VPC peering configuration is the cause of your connection failures and efforts should be focused there.

Sincerely,
James

Ray_Simpson · July 15, 2022, 3:55pm

Hey James,

First thank you for trying to help me, I greatly appreciate it. I went back and recreated everything from epoch and it still failed. However, when I did what you suggested and tried to connect without the private connection - it worked. I have always been of the opinion that the issue is on the cloud side, as I am not a network engineer and have always struggled with that side of technology. I am going to keep working with it and see if I can get it to work over the peer connection. Thank you again.

R

James_Kovacs · July 15, 2022, 6:39pm

Hi, Ray,

Glad to hear that you were able to successfully connect without VPC peering. Yes, network engineering is its own microcosm with specialized skills and nomenclature. And the “magic” of containerized environments introduce their own challenges - especially when the physical NIC is shared between multiple virtual NICs.

Still it is good that you were able to identify that the root cause of the connection failure was network configuration/challenges rather than an intrinsic driver issue. I would encourage you to ask for assistance with VPC peering setup in the MongoDB Atlas category as they will have more experience in this topic.

Sincerely,
James