External Access to Kubernetes Replica Set Fails with MongoNetworkError: getaddrinfo ENOTFOUND

Hi everyone,

I’m having trouble connecting to my MongoDB replica set, which is managed by the MongoDB Kubernetes Operator (mongodb-kubernetes-1.3.0) . I am trying to connect from an external client on my local machine, but the connection fails after the initial handshake.

My Goal

My goal is to successfully connect to the replica set from outside the Kubernetes cluster using mongosh.

The Command I’m Using

I am running mongosh from a Docker container with the following command:
Bash

docker run --name mongotest1 -it --rm alpine/mongosh mongosh “mongodb://m1.infra-dashboards.mobicycle.pt:27017/?replicaSet=mongodb-prod&tls=true” --eval “rs.isMaster();” -u my-user

The Error I Receive

The connection attempt fails with this error, indicating it cannot resolve an internal Kubernetes service name:

Current Mongosh Log ID: 68cc177bbe1d2ccc13bae731
Connecting to: mongodb://@m1.infra-dashboards.mobicycle.pt:27017/?replicaSet=mongodb-prod&tls=true&appName=mongosh+2.0.2
MongoNetworkError: getaddrinfo ENOTFOUND mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local

My Configuration

My current MongoDBCommunity resource is set up for internal cluster access, but I haven’t configured it for external access yet. Here is a simplified version of my deployment YAML:
YAML

apiVersion: MongoDB: The World’s Leading Modern Database | MongoDB
kind: MongoDBCommunity
metadata:
name: mongodb-prod
spec:

Basic replica set configuration

members: 1
version: “8.2.0”
replicaSetHorizons:
- horizon: m1.infra-dashboards.mobicycle.pt:27017

Security and User setup

security:
authentication:
modes: [“SCRAM”]
tls:
enabled: true
certificateKeySecretRef:
name: mongodb-prod-tls-secret

users:
- name: my-user
db: admin
passwordSecretRef:
name: my-user-password # Secret containing the user’s password
roles:
- name: readWriteAnyDatabase
db: admin

rs config

{
_id: ‘mongodb-prod’,
version: 1,
term: 1,
members: [
{
_id: 0,
host: ‘mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local:27017’,
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 1,
tags: {},
horizons: { external: ‘m1.infra-dashboards.mobicycle.pt:27017’ },
secondaryDelaySecs: Long(‘0’),
votes: 1
}
],
protocolVersion: Long(‘1’),
writeConcernMajorityJournalDefault: true,
settings: {
chainingAllowed: true,
heartbeatIntervalMillis: 2000,
heartbeatTimeoutSecs: 10,
electionTimeoutMillis: 10000,
catchUpTimeoutMillis: -1,
catchUpTakeoverDelayMillis: 30000,
getLastErrorModes: {},
getLastErrorDefaults: { w: 1, wtimeout: 0 },
replicaSetId: ObjectId(‘68cc1d464e13c69a9d81b233’)
}

rs isMaster

rs.isMaster()
{
topologyVersion: {
processId: ObjectId(‘68cc1d454e13c69a9d81b22a’),
counter: Long(‘6’)
},
hosts: [ ‘mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local:27017’ ],
setName: ‘mongodb-prod’,
setVersion: 1,
ismaster: true,
secondary: false,
primary: ‘mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local:27017’,
me: ‘mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local:27017’,
electionId: ObjectId(‘7fffffff0000000000000001’),
lastWrite: {
opTime: { ts: Timestamp({ t: 1758209292, i: 1 }), t: Long(‘1’) },
lastWriteDate: ISODate(“2025-09-18T15:28:12.000Z”),
majorityOpTime: { ts: Timestamp({ t: 1758209292, i: 1 }), t: Long(‘1’) },
majorityWriteDate: ISODate(“2025-09-18T15:28:12.000Z”)
},
maxBsonObjectSize: 16777216,
maxMessageSizeBytes: 48000000,
maxWriteBatchSize: 100000,
localTime: ISODate(“2025-09-18T15:28:14.583Z”),
logicalSessionTimeoutMinutes: 30,
connectionId: 359,
minWireVersion: 0,
maxWireVersion: 27,
readOnly: false,
ok: 1,
‘$clusterTime’: {
clusterTime: Timestamp({ t: 1758209292, i: 1 }),
signature: {
hash: Binary.createFromBase64(“WuRFTV4vyUTVwgL+hdzsEcT2L6o=”, 0),
keyId: Long(‘7551442861678395398’)
}
},
operationTime: Timestamp({ t: 1758209292, i: 1 }),
isWritablePrimary: true
}

My Question

The problem is that the primary server reports the pods internal hostname to the client.

How can I correctly modify my MongoDBCommunity resource to advertise publicly resolvable DNS names for members of the replica set? I’ve seen documentation about replicaSetHorizons but would appreciate a clear example of how to configure this properly.

Thank you in advance for your help!

Hey! Welcome to our forums and thanks for a really comprehensive problem statement with all the details included.

The problem I see here is an incomplete external horizon definition in MongoDBCommunity spec:

replicaSetHorizons:
- horizon: m1.infra-dashboards.mobicycle.pt:27017

replicaSetHorizons is a list of map[string]string and it must be populated with as many elements as there are members in the replica set. In your case it should be as follows:

replicaSetHorizons:
- horizon: m1.infra-dashboards.mobicycle.pt:27017
- horizon: m2.infra-dashboards.mobicycle.pt:27017
- horizon: m3.infra-dashboards.mobicycle.pt:27017

should you have another horizon it would become:

replicaSetHorizons:
- horizon: m1.infra-dashboards.mobicycle.pt:27017
  another: m1.another.example.com:27017  
- horizon: m2.infra-dashboards.mobicycle.pt:27017
  another: m2.another.example.com:27017  
- horizon: m3.infra-dashboards.mobicycle.pt:27017
  another: m3.another.example.com:27017  

Please try it out. The reason of having that error message when connecting to “mongodb://m1.infra-dashboards.mobicycle.pt:27017/?replicaSet=mongodb-prod&tls=true” is that the driver in mongosh is correctly connecting to the replicaset, but in the db.hello()/db.isMaster() the other hostnames of the replicaset are returned from the default horizon (internal .cluster.local) as the one defined is incomplete.

Also I’ve noticed that rs.config shows the horizon named “external”, but in the MongoDBCommunity yaml you have it named as “horizon”. Please ensure you have it aligned and perhaps clean up anything added manually to the rs config if that’s the case.

I mistakenly uploaded a previous snippet from the replica set configuration, that is why there was a missmatch between the rs.config and the yaml configuration.
Nevertheless i followed your advice and upped the number of replicas to 3, was previously 1. I configured one horizon for each replica, as it will be possible to see at the end, following that i added TLSRoute to perform SSL Termination at the Gateway level mapping each horizon hostname:port to the appropriate pod.

rs config

I obtained this by using mongosh to connect to the headless mongodb service created by the operator:

{
  _id: 'mongodb-prod',
  version: 1,
  term: 1,
  members: [
    {
      _id: 0,
      host: 'mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      horizons: { horizon: 'm1.infra-dashboards.mobicycle.pt:27017' },
      secondaryDelaySecs: Long('0'),
      votes: 1
    },
    {
      _id: 1,
      host: 'mongodb-prod-1.mongodb-prod-svc.mongodb.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      horizons: { horizon: 'm2.infra-dashboards.mobicycle.pt:27017' },
      secondaryDelaySecs: Long('0'),
      votes: 1
    },
    {
      _id: 2,
      host: 'mongodb-prod-2.mongodb-prod-svc.mongodb.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      horizons: { horizon: 'm3.infra-dashboards.mobicycle.pt:27017' },
      secondaryDelaySecs: Long('0'),
      votes: 1
    }
  ],
  protocolVersion: Long('1'),
  writeConcernMajorityJournalDefault: true,
  settings: {
    chainingAllowed: true,
    heartbeatIntervalMillis: 2000,
    heartbeatTimeoutSecs: 10,
    electionTimeoutMillis: 10000,
    catchUpTimeoutMillis: -1,
    catchUpTakeoverDelayMillis: 30000,
    getLastErrorModes: {},
    getLastErrorDefaults: { w: 1, wtimeout: 0 },
    replicaSetId: ObjectId('68cd2c3da93195c5b8ff0462')
  }
}

just for information:

Name:         mongodb-2-sec-tls
Namespace:    traefik-v2
Labels:       <none>
Annotations:  <none>
API Version:  gateway.networking.k8s.io/v1alpha2
Kind:         TLSRoute
Metadata:
  Creation Timestamp:  2025-09-19T10:21:38Z
  Generation:          3
  Resource Version:    8612052
  UID:                 d9840c8b-9dc3-478f-aea4-b95a48778170
Spec:
  Hostnames:
    m3.infra-dashboards.mobicycle.pt
  Parent Refs:
    Group:         gateway.networking.k8s.io
    Kind:          Gateway
    Name:          traefik-gateway
    Namespace:     traefik-v2
    Section Name:  mongodb3
  Rules:
    Backend Refs:
      Group:
      Kind:       Service
      Name:       mongodb-svc-traefik-mongodb-2
      Namespace:  mongodb
      Port:       27017
      Weight:     1

Test

docker run --name mongotest1 -it --rm alpine/mongosh mongosh "mongodb://m1.infra-dashboards.mobicycle.pt:27017/?replicaSet=mongodb-prod&tls=true" --eval "rs.isMaster();" -u my-user
Enter password: ***************
Current Mongosh Log ID: 68cd2ff54c63a82c555e60f8
Connecting to:          mongodb://<credentials>@m1.infra-dashboards.mobicycle.pt:27017/?replicaSet=mongodb-prod&tls=true&appName=mongosh+2.0.2
MongoNetworkError: getaddrinfo ENOTFOUND mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local

Also performing a packet capture at the node hosting the mongodb master node MONGO Reply states me as the cluster internal name, the same for the hosts section of the reply where there is a list of internal names.

i may not be understanding what is an horizon…
My current hypothesis is that this is a source IP address preservation issue. The MongoDB pods see the connection coming from the internal IP of the Traefik gateway, not from the original external client. Because the source IP is from within the cluster, MongoDB assumes the client is “internal” and correctly provides the internal hostnames. It never uses the external horizons.

Based on this, I have a few questions:

  1. Is my understanding of horizons correct? Is it designed to solve this exact problem, but is failing because the server misidentifies the client’s location?
  2. Is enabling PROXY protocol the right solution? This would allow the gateway to forward the original client’s IP to the MongoDB pods, which should then allow them to provide the correct external hostnames from the horizons configuration.

I am grateful for your fast reply.

Unfortunately replicaSet horizons are pretty confusing in how they work for exposing mongodb nodes externally.

Essentially, the driver connecting to the replicaset needs only one seed node in order to learn about the topology (addresses of the other nodes) of the replicaset using db.hello()/isMaster(). When connecting to the replicaset nodes from inside k8s cluster, there are all nodes visible under the .svc.cluster.local addresses, so nothing to be done here.

But when the driver is connecting to one of the nodes that is exposed externally on a different hostname than the node is internally deployed on, then the db.hello() will return the hostnames of the other nodes as internal addresses.

So that’s why the replicaset horizons mechanism was introduced for. It’s configuring a consistent set of hostnames for a particular networking path the replicaset nodes are exposed on, in order to return a consistent view (a horizon!) of all nodes in the replicaset from the perspective of the client. I hope that makes sense.

And the underlying mechanism that allows mongod to read on which hostname the client is connecting to is the SNI from TLS, which makes TLS mandatory in mongod for horizons to work (but is also quite good to have it when exposing externally).

This diagram should show how the mongod is handling connections from different horizons. I hope it will make it more clear.

              +----------------+
              |     CLIENT     |
              | (Connects with |
              |   TLS + SNI)   |
              +----------------+
                      |
                      +-----> SNI: m3.infra-dashboards.mobicycle.pt
                      |
                      V
              +----------------+
              | MONGOD SERVER  |
              | (Receives TLS  |
              |  conn with SNI)|
              +----------------+
                      |
                      +-----> Looks up hostname from SNI and checks configured horizons
                      |       (the internal hostnames are always part of the __default horizon)
                      |       replicaSetHorizons:
                      |       - __default: mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local:27017
                      |         external:  m1.infra-dashboards.mobicycle.pt:27017
                      |       - __default: mongodb-prod-1.mongodb-prod-svc.mongodb.svc.cluster.local:27017
                      |         external:  m2.infra-dashboards.mobicycle.pt:27017
                      |       - __default: mongodb-prod-2.mongodb-prod-svc.mongodb.svc.cluster.local:27017
                      |         external:  m3.infra-dashboards.mobicycle.pt:27017
                      V
        +-------------------------------------------+
        |         replicaSetHorizons                |
        |            Configuration                  |
        | (Finds that                               |
        |   m3.infra-dashboards.mobicycle.pt:27017  |
        |   is part of "external" horizon)          |
        +-------------------------------------------+
                      |
                      +-----> db.Hello() returns all hosts from "external" horizon
                      |       [
                      |         m1.infra-dashboards.mobicycle.pt:27017,
                      |         m2.infra-dashboards.mobicycle.pt:27017,
                      |         m3.infra-dashboards.mobicycle.pt:27017
                      |       ]
                      V
              +----------------+
              |     CLIENT     |
              | (Connects to   |
              |  all members)  |
              +----------------+

In your case, terminating TLS traffic in Traefik was a mistake as you erased the essential information needed for mongod to get the hostname. Try to configure Traefik in a TLS Passthrough proxy mode without termination. You can probably use SNI field directly in Traefik’s configuration to dispatch the connection to the appropriate pod.

Also I would suggest using a naming convention when exposing externally, to match have the pod name in the external domain. You can then create a semi-automatic dispatching rule in your reverse proxy component of choice and configure it to receive …example.com:27017 and dispatch it to a service’s FQDN using also that pod name.

I hope that helps!

Hello again,

I now understand the requirements to deploy a MongoDB Community replica set with external connectivity. Your guidance was extremely helpful, especially your explanation of how clients retrieve the horizons.

After evaluating the options, I’ve concluded that deploying it this way isn’t feasible under the current constraints of my use case.

I attempted several deployment approaches but encountered obstacles at multiple steps. The most recent issue is the requirement to configure caConfigMap under spec.security.tls. I would prefer using valid certificates generated by cert-manager with ACME solvers, but as far as I can tell, these do not include a ca.crt in the secret, making it impossible to use them in this context. This would leave me with self-signed certificates, which would then need to be distributed to authorized personnel and other services outside of Kubernetes — adding operational overhead I’m trying to avoid.

Once again, thank you for your help. I have no doubt that your solution would work, as I’ve reviewed forums and found no issues related to my situation. The solution made complete sense, so I truly appreciate the time you spent helping me.

Should i mark your last question has :checkered_flag: solution? I did not test it, yet i have no doubt it is the correct way?