Description:
I have configured a mongodb replicaset on a kubernetes cluster (v1.21) with 1 primary and 2 secondary nodes where the client apps (springboot) are successfully able to read/write data to mongodb using the following connection string.
mongodb connection string,
spring.data.mongodb.uri=mongodb://mongo.database.svc:27017/?replicaSet=rs0&readPreference=secondaryPreferred&maxStalenessSeconds=120
Issue:
The client apps fail to reach the mongodb endpoints when the kubernetes cluster restarts (or when the mongodb pods restart) because the client app is trying to connect replicaset using the service DNS that resolves to the old mongodb pod IPaddresses.
Before Restart:
$ oc get po -n database -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mongo-0 2/2 Running 0 2d12h 10.128.2.185 host-node1.novalocal <none> <none>
mongo-1 2/2 Running 0 2d12h 10.128.2.186 host-node1.novalocal <none> <none>
mongo-2 2/2 Running 0 2d12h 10.128.2.187 host-node1.novalocal <none> <none>
After Restart:
$ oc get po -n database -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mongo-0 2/2 Running 0 18s 10.128.3.53 host-node1.novalocal <none> <none>
mongo-1 2/2 Running 0 13s 10.128.3.54 host-node1.novalocal <none> <none>
mongo-2 2/2 Running 0 10s 10.128.3.55 host-node1.novalocal <none> <none>
mongodb connection error from springboot client,
org.springframework.dao.DataAccessResourceFailureException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@6f391070. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.128.2.187:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}, {address=10.128.2.186:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}, {address=10.128.8.138:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches com.mongodb.client.internal.MongoClientDelegate$1@6f391070. Client view of cluster state is {type=REPLICA_SET, servers=[{address=10.128.2.187:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}, {address=10.128.8.136:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}, {address=10.128.8.138:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {java.net.NoRouteToHostException: No route to host (Host unreachable)}}]
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:95)
at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2874)
at org.springframework.data.mongodb.core.MongoTemplate.executeFindOneInternal(MongoTemplate.java:2749)
at org.springframework.data.mongodb.core.MongoTemplate.doFindOne(MongoTemplate.java:2466)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:799)
at org.springframework.data.mongodb.core.MongoTemplate.findOne(MongoTemplate.java:786)
at com.fujitsu.fnc.fums.faultMgmt.service.FaultMgmtService.getNEFormat(FaultMgmtService.java:205)
at com.fujitsu.fnc.fums.faultMgmt.service.FaultMgmtService.getAutonomousFaultResponse(FaultMgmtService.java:129)
at com.fujitsu.fnc.fums.faultMgmt.stream.listener.AutonomousStreamListener.process(AutonomousStreamListener.java:28)
at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source)
Expected Behaviour:
The mongodb service DNS should be properly resolved to the current pod IP addresses.
Actual Behaviour:
The mongodb service DNS are resolving to the old pod IP addresses.
mongodb version,
$ oc exec -it mongo-0 -n database -- mongo --version
Defaulted container "mongo" out of: mongo, mongo-sidecar
MongoDB shell version v4.0.19
git version: 7e28f4296a04d858a2e3dd84a1e79c9ba59a9568
OpenSSL version: OpenSSL 1.0.2g 1 Mar 2016
allocator: tcmalloc
modules: none
build environment:
distmod: ubuntu1604
distarch: x86_64
target_arch: x86_64
OS Details:
$ cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.5 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.5"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.5 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.5
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.5"
Any suggestions would be appreciated.