If one of the members (just a secondary) goes down, then the whole Springboot application goes down, which kind of defeats the purpose of the replica set (if I am not mistaken).
Is there some configuration that I am missing? Either in mongo or Springboot?
Logs ::
rg.springframework.dao.DataAccessResourceFailureException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=ARBITER_HOST:PORT, type=REPLICA_SET_ARBITER, roundTripTime=0.8 ms, state=CONNECTED}, {address=TERTIARY_HOST:PORT, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: TERTIARY_HOST:PORT}, caused by {java.net.ConnectException: Connection refused}}, {address=PRIMARY_HOST:PORT, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoCommandException: Command failed with error 211 (KeyNotFound): ‘Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1609295841, 1) } with id: 6884842166149971971’ on server PRIMARY_HOST:PORT. The full response is { “ok” : 0.0, “errmsg” : “Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1609295841, 1) } with id: 6884842166149971971”, “code” : 211, “codeName” : “KeyNotFound” }}}, {address=SECONDARY_HOST:PORT, type=REPLICA_SET_SECONDARY, roundTripTime=0.9 ms, state=CONNECTED}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=ARBITER_HOST:PORT, type=REPLICA_SET_ARBITER, roundTripTime=0.8 ms, state=CONNECTED}, {address=TERTIARY_HOST:PORT, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: TERTIARY_HOST:PORT}, caused by {java.net.ConnectException: Connection refused}}, {address=PRIMARY_HOST:PORT, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoCommandException: Command failed with error 211 (KeyNotFound): ‘Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1609295841, 1) } with id: 6884842166149971971’ on server PRIMARY_HOST:PORT. The full response is { “ok” : 0.0, “errmsg” : “Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1609295841, 1) } with id: 6884842166149971971”, “code” : 211, “codeName” : “KeyNotFound” }}}, {address=SECONDARY_HOST:PORT, type=REPLICA_SET_SECONDARY, roundTripTime=0.9 ms, state=CONNECTED}]rg.springframework.dao.DataAccessResourceFailureException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=ARBITER_HOST:PORT, type=REPLICA_SET_ARBITER, roundTripTime=0.8 ms, state=CONNECTED}, {address=TERTIARY_HOST:PORT, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: TERTIARY_HOST:PORT}, caused by {java.net.ConnectException: Connection refused}}, {address=PRIMARY_HOST:PORT, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoCommandException: Command failed with error 211 (KeyNotFound): ‘Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1609295841, 1) } with id: 6884842166149971971’ on server PRIMARY_HOST:PORT. The full response is { “ok” : 0.0, “errmsg” : “Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1609295841, 1) } with id: 6884842166149971971”, “code” : 211, “codeName” : “KeyNotFound” }}}, {address=SECONDARY_HOST:PORT, type=REPLICA_SET_SECONDARY, roundTripTime=0.9 ms, state=CONNECTED}]; nested exception is com.mongodb.MongoTimeoutException: Timed out after 30000 ms while waiting for a server that matches ReadPreferenceServerSelector{readPreference=primary}. Client view of cluster state is {type=REPLICA_SET, servers=[{address=ARBITER_HOST:PORT, type=REPLICA_SET_ARBITER, roundTripTime=0.8 ms, state=CONNECTED}, {address=TERTIARY_HOST:PORT, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: TERTIARY_HOST:PORT}, caused by {java.net.ConnectException: Connection refused}}, {address=PRIMARY_HOST:PORT, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoCommandException: Command failed with error 211 (KeyNotFound): ‘Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1609295841, 1) } with id: 6884842166149971971’ on server PRIMARY_HOST:PORT. The full response is { “ok” : 0.0, “errmsg” : “Cache Reader No keys found for HMAC that is valid for time: { ts: Timestamp(1609295841, 1) } with id: 6884842166149971971”, “code” : 211, “codeName” : “KeyNotFound” }}}, {address=SECONDARY_HOST:PORT, type=REPLICA_SET_SECONDARY, roundTripTime=0.9 ms, state=CONNECTED}]
The replica set up is 1P 2S 1A.
If one of the secondaries does down (TERTIARY_HOST in this case), all the spring boot applications fail.
From the excerpt, {address=TERTIARY_HOST:PORT, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSocketOpenException: Exception opening socket}, caused by {io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: TERTIARY_HOST:PORT}, caused by {java.net.ConnectException: Connection refused}}
looks like it continues to connect to the instance even though it is down.