Thank you for your reply @finisky. I tried your suggestion, however, it turns out the issue is different. After commenting here, I found AuthenticationFailed logs under mongodb-agent-monitoring logs. It was working totally fine. I just disabled backup and then ended up with this issue. I am still investigating it.
[2022-02-05T22:40:48.436+0000] [metrics.status.collector-ops-manager-db-2.ops-manager-db-svc.mongodb.svc.cluster.local:27017.error] [monitoring/collector.go:looper:254] Failure during status metrics collection.
Failed to get connectionStatus. Err: `connection() : auth error: sasl conversation error: unable to authenticate using mechanism "SCRAM-SHA-256": (AuthenticationFailed) Authentication failed.`
at cm/monitoring/dialing.go:491
at cm/monitoring/dialing.go:240
at cm/monitoring/dialing.go:351
at cm/monitoring/dialing.go:368
at louisaberger/procexec/concurrency.go:45
at src/runtime/asm_amd64.s:1374
[22:40:48.448] All 0 Mongo processes are in goal state. Monitoring in goal state
Here is my OpsManager manifest if it helps in any way to troubleshoot. I assume this automation user is being created by OpsManager itself, therefore, I don’t know how it was working now but not now.
---
apiVersion: mongodb.com/v1
kind: MongoDBOpsManager
metadata:
name: ops-manager
namespace: mongodb
spec:
version: 4.4.9
configuration:
mms.centralUrl: https://mongodb.mydomain.com
adminCredentials: ops-manager-admin-secret
externalConnectivity:
type: NodePort # TODO: should be service behind nginx ingress
applicationDatabase:
members: 3
version: 4.2.11-ent
persistent: true
podSpec:
persistence:
single:
storage: 100G
storageClass: aws-efs
backup:
enabled: true
opLogStores:
- name: oplog1
mongodbResourceRef:
name: om-mongodb-oplog-db
mongodbUserRef:
name: ops-manager-app
s3Stores:
- name: "news3store"
s3SecretRef:
name: "REDACTED"
s3BucketEndpoint: "REDACTED"
s3BucketName: "REDACTED"
pathStyleAccessEnabled: true
---