Intermittent connection issue between MongoDB pods

Name and Version

Steps will reproduce the bug?
I have two deployments of MongoDB in the same namespace :
a. MongoDB instance to be periodically backed up (mongodb-deployment-79bdf489d8-klyo7)
b. CronJob to do the backups (mongo-backup-toolset-75f8f679dd-abc2d)

NB : CronJob Deployment uses mongosh and thus has a Mongo container also

For backups I will run the following command as part of the CronJob’s logic :

mongodump --uri="mongodb://admin:abc123@mongo-single-clusterip/platforms" --authenticationDatabase admin --gzip --out /tmp/dump-12-06-22/mybackup.gz

Before fully implementing CronJob I first deployed it as a plain MongoDB containerwhy – because I have to test connection between the two pods first – If its authenticating successfully .

I am testing the connection by executing into the CronJob container :

kubectl exec -it mongo-backup-toolset-75f8f679dd-abc2d -- sh

mongo mongodb://admin:abc123@mongo-single-clusterip/platforms?authSource=admin

CronJob definition :


The Deployment to be backed up has the following definition:
MongoDB Manifest

Expected behavior?
A login attempt executed from mongo-backup-toolset-75f8f679dd-abc2d should authenticate successfully and consistently between different sessions . If this is working the corresponding mongodump command should also execute and authenticate consistently since it will be using the same MongoDB-URI and credentials :

mongodump --uri="mongodb://admin:abc123@mongo-single-clusterip/platforms" --authenticationDatabase admin --gzip --out /tmp/dump-12-06-22/mybackup.gz

What is happening?
The connection is intermittent . Sometimes the toolset pod connects/authenticates successfully , sometimes it fails with error :

mongo mongodb://admin:abc123@mongo-single-clusterip/platforms?authSource=admin MongoDB shell version v4.4.6 connecting to: mongodb://mongo-single-clusterip:27017/platforms?authSource=admin&compressors=disabled&gssapiServiceName=mongodb Error: Authentication failed. : connect@src/mongo/shell/mongo.js:374:17 @(connect):2:6 exception: connect failed exiting with code 1

Below is a screenshot of this behavior. I logged in successfully and did a proper logout but authentication fails on the subsequent attempt :