Unable to connect to replica set from k8s cluster

Hello everyone,

I’m trying to deploy a MongoDB cluster on-premise with a community operator. (following the guide here Run Secure Containerized MongoDB Deployments Using the MongoDB Community Kubernetes Operator | MongoDB Blog and from GitHub as well)

All replica sets are deployed and running successfully running. Anyhow, I’m stuck on Connecting to the Replica Set. Getting connection refused error.

ppawar$ kubectl exec -it example-mongodb-0 -n demo -c mongod -- mongo mongodb://example-mongodb-0.example-mongodb-svc.demo.svc.k8s.local:27017 --username my-user --password demopw
MongoDB shell version v4.2.6
connecting to: mongodb://example-mongodb-0.example-mongodb-svc.demo.svc.k8s.local:27017/?compressors=disabled&gssapiServiceName=mongodb
2021-08-28T10:03:09.618+0000 E  QUERY    [js] Error: couldn't connect to server example-mongodb-0.example-mongodb-svc.demo.svc.k8s.local:27017, connection attempt failed: SocketException: Error connecting to mongodb://example-mongodb-0.example-mongodb-svc.demo.svc.k8s.local:27017 (10.111.222.4:27017) :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:341:17
@(connect):2:6
2021-08-28T10:03:09.619+0000 F  -        [main] exception: connect failed
2021-08-28T10:03:09.619+0000 E  -        [main] exiting with code 1
command terminated with exit code 1

Noticeable differences from the guide are I’m unable to see expected output for MONGO_URI.

MONGO_URI="$(kubectl get mdbc example-mongodb -o jsonpath=’{.status.mongoUri}’)"

When I try kubectl get mdbc example-mongodb it outputs below

NAME PHASE VERSION
example-mongodb Running

I tried using the connection string provided for the application access for shell access.

mongodb://example-mongodb-0.example-mongodb-svc.demo.svc.k8s.local:27017

It is a bit different than the one on the guide as on-premise host resolution works for this host.

At this point, I’m not even sure if the mongo server is running. I tried accessing localhost:27017 on mongod container without any luck.

Is it possible that MongoDB isn’t running probably due to the hostname targetted by the mongo operator isn’t being resolved by on-premise setup correctly? Any guidance or pointers would be highly appreciated.

Thank you for taking the time to go through the post.

I did some digging one of the issues encountered while accessing hosted volume was securityContext added on replicasets. Noticed replicasets with securityContext 2000. So I had trouble accessing mounts from within the containers.

Is there a way to set replicaset run all the containers as root?