I mistakenly uploaded a previous snippet from the replica set configuration, that is why there was a missmatch between the rs.config and the yaml configuration.
Nevertheless i followed your advice and upped the number of replicas to 3, was previously 1. I configured one horizon for each replica, as it will be possible to see at the end, following that i added TLSRoute to perform SSL Termination at the Gateway level mapping each horizon hostname:port to the appropriate pod.
rs config
I obtained this by using mongosh to connect to the headless mongodb service created by the operator:
{
_id: 'mongodb-prod',
version: 1,
term: 1,
members: [
{
_id: 0,
host: 'mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local:27017',
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 1,
tags: {},
horizons: { horizon: 'm1.infra-dashboards.mobicycle.pt:27017' },
secondaryDelaySecs: Long('0'),
votes: 1
},
{
_id: 1,
host: 'mongodb-prod-1.mongodb-prod-svc.mongodb.svc.cluster.local:27017',
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 1,
tags: {},
horizons: { horizon: 'm2.infra-dashboards.mobicycle.pt:27017' },
secondaryDelaySecs: Long('0'),
votes: 1
},
{
_id: 2,
host: 'mongodb-prod-2.mongodb-prod-svc.mongodb.svc.cluster.local:27017',
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 1,
tags: {},
horizons: { horizon: 'm3.infra-dashboards.mobicycle.pt:27017' },
secondaryDelaySecs: Long('0'),
votes: 1
}
],
protocolVersion: Long('1'),
writeConcernMajorityJournalDefault: true,
settings: {
chainingAllowed: true,
heartbeatIntervalMillis: 2000,
heartbeatTimeoutSecs: 10,
electionTimeoutMillis: 10000,
catchUpTimeoutMillis: -1,
catchUpTakeoverDelayMillis: 30000,
getLastErrorModes: {},
getLastErrorDefaults: { w: 1, wtimeout: 0 },
replicaSetId: ObjectId('68cd2c3da93195c5b8ff0462')
}
}
just for information:
Name: mongodb-2-sec-tls
Namespace: traefik-v2
Labels: <none>
Annotations: <none>
API Version: gateway.networking.k8s.io/v1alpha2
Kind: TLSRoute
Metadata:
Creation Timestamp: 2025-09-19T10:21:38Z
Generation: 3
Resource Version: 8612052
UID: d9840c8b-9dc3-478f-aea4-b95a48778170
Spec:
Hostnames:
m3.infra-dashboards.mobicycle.pt
Parent Refs:
Group: gateway.networking.k8s.io
Kind: Gateway
Name: traefik-gateway
Namespace: traefik-v2
Section Name: mongodb3
Rules:
Backend Refs:
Group:
Kind: Service
Name: mongodb-svc-traefik-mongodb-2
Namespace: mongodb
Port: 27017
Weight: 1
Test
docker run --name mongotest1 -it --rm alpine/mongosh mongosh "mongodb://m1.infra-dashboards.mobicycle.pt:27017/?replicaSet=mongodb-prod&tls=true" --eval "rs.isMaster();" -u my-user
Enter password: ***************
Current Mongosh Log ID: 68cd2ff54c63a82c555e60f8
Connecting to: mongodb://<credentials>@m1.infra-dashboards.mobicycle.pt:27017/?replicaSet=mongodb-prod&tls=true&appName=mongosh+2.0.2
MongoNetworkError: getaddrinfo ENOTFOUND mongodb-prod-0.mongodb-prod-svc.mongodb.svc.cluster.local
Also performing a packet capture at the node hosting the mongodb master node MONGO Reply states me as the cluster internal name, the same for the hosts section of the reply where there is a list of internal names.
i may not be understanding what is an horizon…
My current hypothesis is that this is a source IP address preservation issue. The MongoDB pods see the connection coming from the internal IP of the Traefik gateway, not from the original external client. Because the source IP is from within the cluster, MongoDB assumes the client is “internal” and correctly provides the internal hostnames. It never uses the external horizons.
Based on this, I have a few questions:
- Is my understanding of
horizonscorrect? Is it designed to solve this exact problem, but is failing because the server misidentifies the client’s location? - Is enabling PROXY protocol the right solution? This would allow the gateway to forward the original client’s IP to the MongoDB pods, which should then allow them to provide the correct external hostnames from the
horizonsconfiguration.
I am grateful for your fast reply.