Mongodb unreachable when its service DNS resolves to old IP addresses of the mongodb pods after the kubernetes cluster restart

Description:
I have configured a mongodb replicaset on a kubernetes cluster (v1.21) with 1 primary and 2 secondary nodes where the client apps (springboot) are successfully able to read/write data to mongodb using the following connection string.

mongodb connection string,

spring.data.mongodb.uri=mongodb://mongo.database.svc:27017/?replicaSet=rs0&readPreference=secondaryPreferred&maxStalenessSeconds=120

Issue:
The client apps fail to reach the mongodb endpoints when the kubernetes cluster restarts (or when the mongodb pods restart) because the client app is trying to connect replicaset using the service DNS that resolves to the old mongodb pod IPaddresses.

Before Restart:

$ oc get po -n database -o wide
NAME                                 READY   STATUS    RESTARTS   AGE     IP             NODE                  NOMINATED NODE   READINESS GATES
mongo-0                              2/2     Running   0          2d12h   10.128.2.185   host-node1.novalocal   <none>           <none>
mongo-1                              2/2     Running   0          2d12h   10.128.2.186   host-node1.novalocal   <none>           <none>
mongo-2                              2/2     Running   0          2d12h   10.128.2.187   host-node1.novalocal   <none>           <none>

After Restart:

$ oc get po -n database -o wide
NAME                                 READY   STATUS    RESTARTS   AGE   IP            NODE                  NOMINATED NODE   READINESS GATES
mongo-0                              2/2     Running   0          18s   10.128.3.53   host-node1.novalocal   <none>           <none>
mongo-1                              2/2     Running   0          13s   10.128.3.54   host-node1.novalocal   <none>           <none>
mongo-2                              2/2     Running   0          10s   10.128.3.55   host-node1.novalocal   <none>           <none>

mongodb connection error from springboot client,

org.springframework.dao.DataAccessResourceFailureException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@6f391070. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.128.2.187:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}, {address=10.128.2.186:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}, {address=10.128.8.138:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@6f391070. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.128.2.187:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}, {address=10.128.8.136:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}, {address=10.128.8.138:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}]
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:95)
at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2874)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindOneInternal(MongoTemplate.java:2749)
at org.springframework.data.mongodb.core.MongoTemplate.doFindOne(MongoTemplate.java:2466)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:799)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:786)
at com.fujitsu.fnc.fums.faultMgmt.service.FaultMgmtService.getNEFormat(FaultMgmtService.java:205)
at com.fujitsu.fnc.fums.faultMgmt.service.FaultMgmtService.getAutonomousFaultResponse(FaultMgmtService.java:129)
at com.fujitsu.fnc.fums.faultMgmt.stream.listener.AutonomousStreamListener.process(AutonomousStreamListener.java:28)
at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source)

Expected Behaviour:
The mongodb service DNS should be properly resolved to the current pod IP addresses.

Actual Behaviour:
The mongodb service DNS are resolving to the old pod IP addresses.

mongodb version,

$ oc exec -it mongo-0 -n database -- mongo --version
Defaulted container "mongo" out of: mongo, mongo-sidecar
MongoDB shell version v4.0.19
git version: 7e28f4296a04d858a2e3dd84a1e79c9ba59a9568
OpenSSL version: OpenSSL 1.0.2g  1 Mar 2016
allocator: tcmalloc
modules: none
build environment:
    distmod: ubuntu1604
    distarch: x86_64
    target_arch: x86_64

OS Details:

$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.5 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.5 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.5"

Any suggestions would be appreciated.

Hi @bhavaniprasad_reddy and welcome to the community!!

Could you help by providing the following details for the above mentioned issue:

  1. How is the application connected to the database in the kubernetes?
  2. The deployment.yaml file
  3. The liveliness and readiness probe values if any
  4. Does this issue happen with every restart?
  5. Logs from when the pod restarted from start to when the issues seems to occur.

Also, as I understand from the version mentioned, you are using MongoDB 4.0 which is no longer supported and hence no further updates will be available for it. Please refer to the following documentation to understand the next available and supported versions and upgrades.

Please help us with the above information so that we could assist you further.

Thanks
Aasawari