Connect issue from local to private MongoDB replica instances via public bastion

HI

I want to ask how I solve this problem.

currently I run mongoDB on AWS (EC2) and I configured replica set

(which means I have 3 instances, primary, secondary, abiter, actually the configuration was successful

let me describe our AWS architecture

We have a Bastion instance on public that facing global internet(just for ssh tunneling)

and as I mentioned We have 3 EC2 instances what running mongoDB replica on private subnet

so, when I tried to connect from local(which means from global) I have to go to Bastion instance first and then connect to private MongoDB instance

but, "No replica set members found yet, Timeout: 10.0s, Topology Description: <TopologyDescription id: 6333aa0a4f0ddb9c168e0143, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription (‘10.1.3.108’, 27017) server_type: Unknown, rtt: None>, <ServerDescription (‘10.1.3.43’, 27017) server_type: Unknown, rtt: None>, <ServerDescription (‘10.1.3.49’, 27017) server_type: Unknown, rtt: None>]>
"
this error message came out first

because I connected to mongoDB by using nginx reverse proxy

In Bastion that facing public,
/etc/nginx/nginx.conf

stream {
    server {
           listen 27018;
           proxy_pass 10.1.3.108:27017;
        }
}

I add this to redirect to actual mongoDB instance(primary)

and then In applicaton

self.client = MongoClient(
                           host=Bastion's public IP,
                           port=27018,
                           username=settings.MONGO_USERNAME,
                           password=settings.MONGO_PASSWORD,
                           authSource=settings.MONGO_AUTH_SOURCE,
                           authMechanism=settings.MONGO_AUTH_MECHANISM,
                           serverSelectionTimeoutMS=10000
                          )

at the log I found It reached successfuly first,

I think after first touch, the primary mongodb instance return replica member’s name which I configured first to client

after first touch client tried to connect private IP, but cilent couldn’t reach to private IP

that’s why this situation happened I guess

and then I tried to connect via ssh tunneling In python app

self.client = MongoSession(
                        host=Bastion's public IP,
                        port=22,
                        user='ec2-user',
                        key=key_path,
                        to_port=27017,
                        to_host='10.1.3.108'
                )

but this time It didn’t find internal host ‘10.1.3.108’

10.1.3.108:56424: timed out, Timeout: 30s, Topology Description: <TopologyDescription id: 6333c051c87d04dc997fdc49, topology_type: Unknown, servers: [<ServerDescription ('10.1.3.108', 56424) server_type: Unknown, rtt: None, error=NetworkTimeout('10.1.3.108:56424: timed out')>]>

actually It doesn’t affect to actual service operation,

this problem makes coworker couldn’t test on local.

I know our test environment sucks

But I can’t help

plz give me solution

thanks

P.S It didn’t any happen when we use mongoDB standalone

self.client = MongoSession(
                        host=bastion's public IP,
                        port=22,
                        user='ec2-user',
                        key=key_path,
                        to_port=27018,
                        to_host='127.0.0.1'
                )

even I modify like this

I got

10.1.3.49:27017: timed out,10.1.3.108:27017: timed out,10.1.3.43:27017: timed out, Timeout: 30s, Topology Description: <TopologyDescription id: 633414a21bc35067764db51b, topology_type: ReplicaSetNoPrimary, servers: [<ServerDescription ('10.1.3.108', 27017) server_type: Unknown, rtt: None, error=NetworkTimeout('10.1.3.108:27017: timed out')>, <ServerDescription ('10.1.3.43', 27017) server_type: Unknown, rtt: None, error=NetworkTimeout('10.1.3.43:27017: timed out')>, <ServerDescription ('10.1.3.49', 27017) server_type: Unknown, rtt: None, error=NetworkTimeout('10.1.3.49:27017: timed out')>]>

this

and I could see

{"t":{"$date":"2022-09-28T09:32:18.758+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.1.1.99:60790","uuid":"54bf27c6-99ef-4d5f-afda-53e4187f1e68","connectionId":101,"connectionCount":15}}
{"t":{"$date":"2022-09-28T09:32:18.768+00:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn101","msg":"client metadata","attr":{"remote":"10.1.1.99:60790","client":"conn101","doc":{"driver":{"name":"PyMongo","version":"4.1.1"},"os":{"type":"Darwin","name":"Darwin","architecture":"x86_64","version":"10.16"},"platform":"CPython 3.7.9.final.0"}}}
{"t":{"$date":"2022-09-28T09:32:18.779+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn101","msg":"Connection ended","attr":{"remote":"10.1.1.99:60790","uuid":"54bf27c6-99ef-4d5f-afda-53e4187f1e68","connectionId":101,"connectionCount":14}}
{"t":{"$date":"2022-09-28T09:32:18.788+00:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"10.1.1.99:60794","uuid":"cba8ccb4-1353-4b41-9679-795484e7b062","connectionId":102,"connectionCount":15}}
{"t":{"$date":"2022-09-28T09:32:18.796+00:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn102","msg":"Connection ended","attr":{"remote":"10.1.1.99:60794","uuid":"cba8ccb4-1353-4b41-9679-795484e7b062","connectionId":102,"connectionCount":14}}

this log,
It reached primary Mongodb instance which is located in private network once
(10.1.1.99 < this is bastion private ip located In public)

Hello @williams3443 ,

Welcome to The MongoDB Community Forums!

If I understand correctly, the bastion connection was setup to allow a connection only to 10.1.3.108:27017 however the replica set are using 10.1.3.49:27017, 10.1.3.108:27017, 10.1.3.43:27017 is this correct? Have you been successful in making the expected connections before, or it never succeed due to the bastion setup?

MongoDB official drivers follow this spec for monitoring the state of all nodes in the replica set. This means that the driver must be able to connect to all nodes in the replica set. Connecting and monitoring to all nodes in a replica set is a necessity, since a replica set provides high availability. If the primary goes down, the driver needs to be able to automatically switch to the new primary. This would not be possible unless the driver can connect to all members.

Regards,
Tarun

First thank you for replying @Tarun_Gaur ,

actually I already solved this problem by using VPN service,

this problem was a little complicated

after first touch to primary via bastion host, It returned replica’s private IP

then client( in this case my local) tried to connect private IP again So I couldn’t reach

but after using VPN. client(local) can connect to private VPC subnet

anyway really thanks to your reply,

Regards,
YoungHoon

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.