Hello,
I was trying to adjust replicas number in the CR MongoDB yaml manifest….
**Only change** to current CR was changing **'replicas' from 2 to 3**.
That is strange because according to operator I should be able to do this. My statefulset is not scaling. I checked last applied configurations, last success apply and no any differences were there. I compared these configuration to my VCS configuration. **No changes were detected**.
Using **6.0.4 MongoDB community version**.
Using **0.7.8 operator version**.
I tried removing annotations for existing CRD which states as 'failed' after apply and reapply again - no result. Nothing was changed so previous setup is still online and working. However, it prompts errors on operator without any reasons even after applying same configuration with 2 'replicas' again.
I have another environment with same specific operator and MongoDB versions. I was able to add replica and even arbiter to specs there. That was also the **only** change made to CR of MongoDB.
**_The error I get_**:
`ERROR controllers/mongodb_status_options.go:104 Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps “mongodb” is invalid: spec: Forbidden: updates to statefulset spec for fields other than ‘replicas’, ‘ordinals’, ‘template’, ‘updateStrategy’, ‘persistentVolumeClaimRetentionPolicy’ and ‘minReadySeconds’ are forbidden`
**What did you do to encounter the bug?**
Steps to reproduce the behavior:
1. Changed my CR MongoDB cluster yaml manifest _database.yaml_ `spec.members: 2` to `spec.members: 3`
2. Use `kubectl apply -f database.yaml`
3. Checked MongoDB statefulset pods (no replica was added).
4. Checked MongoDB community operator logs. Observed the error.
**What did you expect?**
I expected operator to add an additional member to existing MongoDB cluster in statefulset. Making members number 3 instead of existing 2.
**What happened instead?**
Statefulset of MongoDB still had 2 members. MongoDB operator threw error (pasted in description).
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Operator Information**
- 0.7.8
- docker.io/mongo:6.0.4
**Kubernetes Cluster Information**
- AWS EKS
- 1.28
If possible, please include:
- The operator logs:
`sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227
2024-09-04T07:11:25.133Z INFO controllers/replica_set_controller.go:137 Reconciling MongoDB {"ReplicaSet": "mongodb-<NAME>/mongodb"}
2024-09-04T07:11:25.134Z DEBUG controllers/replica_set_controller.go:139 Validating MongoDB.Spec {"ReplicaSet": "mongodb-<NAME>/mongodb"}
2024-09-04T07:11:25.134Z DEBUG controllers/replica_set_controller.go:148 Ensuring the service exists {"ReplicaSet": "mongodb-<NAME>/mongodb"}
2024-09-04T07:11:25.134Z DEBUG agent/replica_set_port_manager.go:122 No port change required {"ReplicaSet": "mongodb-<NAME>/mongodb"}
2024-09-04T07:11:25.142Z INFO controllers/replica_set_controller.go:463 Create/Update operation succeeded {"ReplicaSet": "mongodb-<NAME>/mongodb", "operation": "updated"}
2024-09-04T07:11:25.142Z DEBUG controllers/replica_set_controller.go:409 Scaling up the ReplicaSet, the StatefulSet must be updated first {"ReplicaSet": "mongodb-<NAME>/mongodb"}
2024-09-04T07:11:25.142Z INFO controllers/replica_set_controller.go:330 Creating/Updating StatefulSet {"ReplicaSet": "mongodb-<NAME>/mongodb"}
2024-09-04T07:11:25.151Z ERROR controllers/mongodb_status_options.go:104 Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps "mongodb" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
github.com/mongodb/mongodb-kubernetes-operator/controllers.messageOption.ApplyOption
/workspace/controllers/mongodb_status_options.go:104
github.com/mongodb/mongodb-kubernetes-operator/pkg/util/status.Update
/workspace/pkg/util/status/status.go:25
github.com/mongodb/mongodb-kubernetes-operator/controllers.ReplicaSetReconciler.Reconcile
/workspace/controllers/replica_set_controller.go:200
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.2/pkg/internal/controller/controller.go:227`
- Below we assume that your replicaset database pods are named `mongo-<>`. For instance:
```
❯ k get pods
NAME READY STATUS RESTARTS AGE
mongo-0 2/2 Running 0 19h
mongo-1 2/2 Running 0 19h
❯ k get mdbc
NAME PHASE VERSION
mongo Running 4.4.0
```
- yaml definitions of your MongoDB Deployment(s):
- `kubectl get mdbc -oyaml`
```
apiVersion: v1
items:
- apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"mongodbcommunity.mongodb.com/v1","kind":"MongoDBCommunity","metadata":{"annotations":{},"name":"mongodb","namespace":"mongodb-<SENSITIVE>"},"spec":{"additionalMongodConfig":{"storage.wiredTiger.engineConfig.journalCompressor":"zlib"},"members":3,"security":{"authentication":{"modes":["SCRAM"]}},"statefulSet":{"spec":{"template":{"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"NodeGroup","operator":"In","values":["<SENSITIVE>"]}]}]}},"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchExpressions":[{"key":"app","operator":"In","values":["mongodb"]}]},"topologyKey":"kubernetes.io/hostname"},"weight":100}]}},"containers":[{"name":"mongod","resources":{"limits":{"cpu":"1","memory":"2Gi"},"requests":{"cpu":"500m","memory":"1Gi"}}}]}},"volumeClaimTemplates":[{"metadata":{"name":"data-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"70G"}},"storageClassName":"ebs-sc"}},{"metadata":{"name":"logs-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10G"}},"storageClassName":"ebs-sc"}}]}},"type":"ReplicaSet","users":[{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"readWrite"},{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"readWrite"},{"db":"<SENSITIVE>","name":"readWrite"},{"db":"<SENSITIVE>","name":"readWrite"}],"scramCredentialsSecretName":"<SENSITIVE>"},{"db":"<SENSITIVE>","name":"<SENSITIVE>","passwordSecretRef":{"name":"<SENSITIVE>"},"roles":[{"db":"<SENSITIVE>","name":"read"},{"db":"<SENSITIVE>","name":"read"},{"db":"<SENSITIVE>","name":"changestreamrole"}],"scramCredentialsSecretName":"<SENSITIVE>"}],"version":"6.0.4"}}
mongodb.com/v1.lastAppliedMongoDBVersion: 6.0.4
creationTimestamp: "2024-01-03T07:47:03Z"
generation: 18
labels:
k8slens-edit-resource-version: v1
name: mongodb
namespace: mongodb-<SENSITIVE>
resourceVersion: "386378973"
uid: 8dbc92a1-061b-4ebb-a2be-d1b5dd6d696b
spec:
additionalMongodConfig:
storage.wiredTiger.engineConfig.journalCompressor: zlib
members: 3
security:
authentication:
ignoreUnknownUsers: true
modes:
- SCRAM
statefulSet:
spec:
template:
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: NodeGroup
operator: In
values:
- <SENSITIVE>
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- name: mongod
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 500m
memory: 1Gi
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 70G
storageClassName: ebs-sc
- metadata:
name: logs-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10G
storageClassName: ebs-sc
type: ReplicaSet
users:
- db: <SENSITIVE>
name: <SENSITIVE>
passwordSecretRef:
name: <SENSITIVE>
roles:
- db: <SENSITIVE>
name: clusterAdmin
- db: <SENSITIVE>
name: userAdminAnyDatabase
scramCredentialsSecretName: <SENSITIVE>
- db: <SENSITIVE>
name: <SENSITIVE>
passwordSecretRef:
name: <SENSITIVE>
roles:
- db: <SENSITIVE>
name: readWrite
scramCredentialsSecretName: <SENSITIVE>
- db: <SENSITIVE>
name: <SENSITIVE>
passwordSecretRef:
name: <SENSITIVE>
roles:
- db: <SENSITIVE>
name: readWrite
scramCredentialsSecretName: <SENSITIVE>
- db: <SENSITIVE>
name: <SENSITIVE>
passwordSecretRef:
name: <SENSITIVE>
roles:
- db: <SENSITIVE>
name: readWrite
- db: <SENSITIVE>
name: readWrite
scramCredentialsSecretName: backup-user-scram
- db: <SENSITIVE>
name: <SENSITIVE>
passwordSecretRef:
name: <SENSITIVE>
roles:
- db: <SENSITIVE>
name: dbAdmin
- db: <SENSITIVE>
name: userAdminAnyDatabase
- db: <SENSITIVE>
name: readWrite
- db: <SENSITIVE>
name: readWrite
- db: <SENSITIVE>
name: readWrite
scramCredentialsSecretName: <SENSITIVE>
- db: <SENSITIVE>
name: <SENSITIVE>
passwordSecretRef:
name: <SENSITIVE>
roles:
- db: <SENSITIVE>
name: read
- db: <SENSITIVE>
name: read
- db: <SENSITIVE>
name: changestreamrole
scramCredentialsSecretName: <SENSITIVE>
version: 6.0.4
status:
currentMongoDBMembers: 2
currentStatefulSetReplicas: 2
message: 'Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet:
error creating/updating StatefulSet: StatefulSet.apps "mongodb" is invalid:
spec: Forbidden: updates to statefulset spec for fields other than ''replicas'',
''ordinals'', ''template'', ''updateStrategy'', ''persistentVolumeClaimRetentionPolicy''
and ''minReadySeconds'' are forbidden'
mongoUri: <SENSITIVE>,<SENSITIVE>/?replicaSet=mongodb
phase: Failed
version: 6.0.4
kind: List
metadata:
resourceVersion: ""
```
- yaml definitions of your kubernetes objects like the statefulset(s), pods (we need to see the state of the containers):
- `kubectl get sts -oyaml`
```
apiVersion: v1
items:
- apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: "2024-01-03T07:47:03Z"
generation: 31
labels:
app: mongodb-<SENSITIVE>
name: mongodb
namespace: mongodb-<SENSITIVE>
ownerReferences:
- apiVersion: mongodbcommunity.mongodb.com/v1
blockOwnerDeletion: true
controller: true
kind: MongoDBCommunity
name: mongodb
uid: <SENSITIVE>
resourceVersion: "<SENSITIVE>"
uid: <SENSITIVE>
spec:
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Retain
podManagementPolicy: OrderedReady
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: mongodb-<SENSITIVE>
serviceName: mongodb-<SENSITIVE>
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2024-09-02T08:55:35Z"
creationTimestamp: null
labels:
app: mongodb-<SENSITIVE>
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: NodeGroup
operator: In
values:
- <SENSITIVE>
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- command:
- /bin/sh
- -c
- |2+
#run post-start hook to handle version changes
/hooks/version-upgrade
# wait for config and keyfile to be created by the agent
while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;
# start mongod with this configuration
exec mongod -f /data/automation-mongod.conf;
env:
- name: AGENT_STATUS_FILEPATH
value: /healthstatus/agent-health-status.json
image: docker.io/mongo:6.0.4
imagePullPolicy: IfNotPresent
name: mongod
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 500m
memory: 1Gi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: data-volume
- mountPath: /healthstatus
name: healthstatus
- mountPath: /hooks
name: hooks
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- command:
- /bin/bash
- -c
- |-
current_uid=$(id -u)
AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -maxLogFileDurationHrs 24 -logLevel DEBUG
env:
- name: AGENT_STATUS_FILEPATH
value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
- name: AUTOMATION_CONFIG_MAP
value: mongodb-config
- name: HEADLESS_AGENT
value: "true"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1
imagePullPolicy: Always
name: mongodb-agent
readinessProbe:
exec:
command:
- /opt/scripts/readinessprobe
failureThreshold: 40
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/lib/automation/config
name: automation-config
readOnly: true
- mountPath: /data
name: data-volume
- mountPath: /var/log/mongodb-mms-automation/healthstatus
name: healthstatus
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
dnsPolicy: ClusterFirst
initContainers:
- command:
- cp
- version-upgrade-hook
- /hooks/version-upgrade
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6
imagePullPolicy: Always
name: mongod-posthook
resources: {}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /hooks
name: hooks
- command:
- cp
- /probes/readinessprobe
- /opt/scripts/readinessprobe
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12
imagePullPolicy: Always
name: mongodb-agent-readinessprobe
resources: {}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: 2000
runAsNonRoot: true
runAsUser: 2000
serviceAccount: mongodb-<SENSITIVE>
serviceAccountName: mongodb-<SENSITIVE>
terminationGracePeriodSeconds: 30
volumes:
- emptyDir: {}
name: agent-scripts
- name: automation-config
secret:
defaultMode: 416
secretName: mongodb-<SENSITIVE>
- emptyDir: {}
name: healthstatus
- emptyDir: {}
name: hooks
- emptyDir: {}
name: mongodb-<SENSITIVE>
- emptyDir: {}
name: tmp
updateStrategy:
type: RollingUpdate
volumeClaimTemplates:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: data-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 50G
storageClassName: ebs-sc
volumeMode: Filesystem
status:
phase: Pending
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
creationTimestamp: null
name: logs-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10G
storageClassName: ebs-sc
volumeMode: Filesystem
status:
phase: Pending
status:
availableReplicas: 2
collisionCount: 0
currentReplicas: 2
currentRevision: mongodb-<SENSITIVE>
observedGeneration: 31
readyReplicas: 2
replicas: 2
updateRevision: mongodb-<SENSITIVE>
updatedReplicas: 2
kind: List
metadata:
resourceVersion: ""
```
- `kubectl get pods -oyaml`
```
apiVersion: v1
items:
- apiVersion: v1
kind: Pod
metadata:
annotations:
agent.mongodb.com/version: "6"
kubectl.kubernetes.io/restartedAt: "2024-09-02T08:55:35Z"
creationTimestamp: "2024-09-02T08:56:31Z"
generateName: mongodb-
labels:
app: mongodb-<SENSITIVE>
apps.kubernetes.io/pod-index: "0"
controller-revision-hash: mongodb-<SENSITIVE>
statefulset.kubernetes.io/pod-name: <SENSITIVE>
name: mongodb-0
namespace: mongodb-<SENSITIVE>
ownerReferences:
- apiVersion: apps/v1
blockOwnerDeletion: true
controller: true
kind: StatefulSet
name: mongodb
uid: 934952c3-22f8-4b93-a6e3-34c85bc85fc0
resourceVersion: "386287077"
uid: 5c442610-203d-45ca-b280-dec81b9a923d
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: NodeGroup
operator: In
values:
- <SENSITIVE>
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- mongodb
topologyKey: kubernetes.io/hostname
weight: 100
containers:
- command:
- /bin/sh
- -c
- |2+
#run post-start hook to handle version changes
/hooks/version-upgrade
# wait for config and keyfile to be created by the agent
while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;
# start mongod with this configuration
exec mongod -f /data/automation-mongod.conf;
env:
- name: AGENT_STATUS_FILEPATH
value: /healthstatus/agent-health-status.json
image: docker.io/mongo:6.0.4
imagePullPolicy: IfNotPresent
name: mongod
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 500m
memory: 1Gi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /data
name: <SENSITIVE>
- mountPath: /healthstatus
name: healthstatus
- mountPath: /hooks
name: hooks
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: <SENSITIVE>
- mountPath: /tmp
name: <SENSITIVE>
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-t6sf8
readOnly: true
- command:
- /bin/bash
- -c
- |-
current_uid=$(id -u)
AGENT_API_KEY="$(cat /mongodb-automation/agent-api-key/agentApiKey)"
declare -r current_uid
if ! grep -q "${current_uid}" /etc/passwd ; then
sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd
echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd
export NSS_WRAPPER_PASSWD=/tmp/passwd
export LD_PRELOAD=libnss_wrapper.so
export NSS_WRAPPER_GROUP=/etc/group
fi
agent/mongodb-agent -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json -serveStatusPort=5000 -cluster=/var/lib/automation/config/cluster-config.json -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile /var/log/mongodb-mms-automation/automation-agent.log -maxLogFileDurationHrs 24 -logLevel DEBUG
env:
- name: AGENT_STATUS_FILEPATH
value: /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
- name: AUTOMATION_CONFIG_MAP
value: mongodb-config
- name: HEADLESS_AGENT
value: "true"
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1
imagePullPolicy: Always
name: mongodb-agent
readinessProbe:
exec:
command:
- /opt/scripts/readinessprobe
failureThreshold: 40
initialDelaySeconds: 5
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 500M
requests:
cpu: 500m
memory: 400M
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/lib/automation/config
name: automation-config
readOnly: true
- mountPath: /data
name: data-volume
- mountPath: /var/log/mongodb-mms-automation/healthstatus
name: healthstatus
- mountPath: /var/log/mongodb-mms-automation
name: logs-volume
- mountPath: /var/lib/mongodb-mms-automation/authentication
name: mongodb-keyfile
- mountPath: /tmp
name: tmp
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-t6sf8
readOnly: true
dnsPolicy: ClusterFirst
enableServiceLinks: true
hostname: <SENSITIVE>
initContainers:
- command:
- cp
- version-upgrade-hook
- /hooks/version-upgrade
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6
imagePullPolicy: Always
name: mongod-posthook
resources: {}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /hooks
name: hooks
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-t6sf8
readOnly: true
- command:
- cp
- /probes/readinessprobe
- /opt/scripts/readinessprobe
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12
imagePullPolicy: Always
name: mongodb-agent-readinessprobe
resources: {}
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /opt/scripts
name: agent-scripts
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-t6sf8
readOnly: true
nodeName: <SENSITIVE>
preemptionPolicy: PreemptLowerPriority
priority: 0
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
fsGroup: <SENSITIVE>
runAsNonRoot: true
runAsUser: <SENSITIVE>
serviceAccount: <SENSITIVE>
serviceAccountName: <SENSITIVE>
subdomain: <SENSITIVE>
terminationGracePeriodSeconds: 30
tolerations:
- effect: NoExecute
key: node.kubernetes.io/not-ready
operator: Exists
tolerationSeconds: 300
- effect: NoExecute
key: node.kubernetes.io/unreachable
operator: Exists
tolerationSeconds: 300
volumes:
- name: data-volume
persistentVolumeClaim:
claimName: data-volume-mongodb-0
- name: logs-volume
persistentVolumeClaim:
claimName: logs-volume-mongodb-0
- emptyDir: {}
name: agent-scripts
- name: automation-config
secret:
defaultMode: 416
secretName: <SENSITIVE>
- emptyDir: {}
name: healthstatus
- emptyDir: {}
name: hooks
- emptyDir: {}
name: <SENSITIVE>
- emptyDir: {}
name: tmp
- name: kube-api-access-t6sf8
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: <SENSITIVE>
- configMap:
items:
- key: <SENSITIVE>
path: <SENSITIVE>
name: <SENSITIVE>
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
status:
conditions:
- lastProbeTime: null
lastTransitionTime: "2024-09-02T08:56:38Z"
status: "True"
type: Initialized
- lastProbeTime: null
lastTransitionTime: "2024-09-02T08:56:55Z"
status: "True"
type: Ready
- lastProbeTime: null
lastTransitionTime: "2024-09-02T08:56:55Z"
status: "True"
type: ContainersReady
- lastProbeTime: null
lastTransitionTime: "2024-09-02T08:56:31Z"
status: "True"
type: PodScheduled
containerStatuses:
- containerID: containerd://5a01d0a5116ac5af31cee1f7b0fe6b516d22b5e223042adfb035ee3fcaf7d284
image: docker.io/library/mongo:6.0.4
imageID: docker.io/library/mongo@sha256:a4f2db6f54aeabba562cd07e5cb758b55d6192dcc6f36322a334ba0b0120aaf1
lastState: {}
name: mongod
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-09-02T08:56:38Z"
- containerID: containerd://fab2952924c2028fab4117bb26aaf53715a1993792a5730226823925fb9146db
image: quay.io/mongodb/mongodb-agent:12.0.15.7646-1
imageID: quay.io/mongodb/mongodb-agent@sha256:00dc01552af1eac8a457d7214ae4f913dd2a14409c0125dead25808e40cc0d62
lastState: {}
name: mongodb-agent
ready: true
restartCount: 0
started: true
state:
running:
startedAt: "2024-09-02T08:56:38Z"
hostIP: <SENSITIVE>
initContainerStatuses:
- containerID: containerd://42c48fb8a9556f01b3ab8d61b985aec62c4e177dc53b2b65e0afecaa3991dbe0
image: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.6
imageID: quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook@sha256:5c1483fa22cfb772f186a30180fcfa4af91ff3a638b28a37cc9f997f8ac046f9
lastState: {}
name: mongod-posthook
ready: true
restartCount: 0
started: false
state:
terminated:
containerID: containerd://42c48fb8a9556f01b3ab8d61b985aec62c4e177dc53b2b65e0afecaa3991dbe0
exitCode: 0
finishedAt: "2024-09-02T08:56:36Z"
reason: Completed
startedAt: "2024-09-02T08:56:36Z"
- containerID: containerd://144d11be92c06da8c8c69faa841708f4caf73ebb9a9255cecfa499e8a05befa4
image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.12
imageID: quay.io/mongodb/mongodb-kubernetes-readinessprobe@sha256:ef6c144de962bc976161b41972cc85988189870e05ad4ada0798200c0a988547
lastState: {}
name: mongodb-agent-readinessprobe
ready: true
restartCount: 0
started: false
state:
terminated:
containerID: containerd://144d11be92c06da8c8c69faa841708f4caf73ebb9a9255cecfa499e8a05befa4
exitCode: 0
finishedAt: "2024-09-02T08:56:37Z"
reason: Completed
startedAt: "2024-09-02T08:56:37Z"
phase: Running
podIP: <SENSITIVE>
podIPs:
- ip: <SENSITIVE>
qosClass: Burstable
startTime: "2024-09-02T08:56:31Z"
kind: List
metadata:
resourceVersion: ""
```
- The agent clusterconfig of the faulty members:
- `kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/lib/automation/config/cluster-config.json`
```
{"version":6,"processes":[{"name":"mongodb-0","disabled":false,"hostname":"<SENSITIVE>","args2_6":{"net":{"port":27017},"repli
cation":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilityVersion":"6.0","processTyp
e":"mongod","version":"6.0.4","authSchemaVersion":5},{"name":"mongodb-1","disabled":false,"hostname":"<SENSITIVE>","args2_6":{
"net":{"port":27017},"replication":{"replSetName":"mongodb"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}},"featureCompatibilit
yVersion":"6.0","processType":"mongod","version":"6.0.4","authSchemaVersion":5}],"replicaSets":[{"_id":"mongodb","members":[{"_id":0,"host":"mongodb-0","priority":1,"arbi
terOnly":false,"votes":1},{"_id":1,"host":"mongodb-1","priority":1,"arbiterOnly":false,"votes":1}],"protocolVersion":"1","numberArbiters":0}],"auth":{"usersWanted":[{"mec
hanisms":[],"roles":[{"role":"clusterAdmin","<SENSITIVE>":"<SENSITIVE>"},{"role":"userAdminAnyDatabase","<SENSITIVE>":"<SENSITIVE>"}],"user":"<SENSITIVE>","<SENSITIVE>":"<SENSITIVE>","authenticationRestrictions":[],"
scramSha256Creds":{"iterationCount":15000,"salt":"WioeMJQXT8w9Coif5Fq3gqV1OfeDi64Bvq/maw==","serverKey":"iK8SJLwCzmk95+mUePC3wrpGw29Tfx9vN+ZCKSMPKMM=","storedKey":"Z33GU9
ix2W++nlnkFBIbYP7kEATZ/6sDVQqdhEd+tT0="},"scramSha1Creds":{"iterationCount":10000,"salt":"Q9mmbNXpyLDRtYmoln1xgA==","serverKey":"AmaNP+YmbrNf23l8URaZAZKKOz0=","storedKey"
:"0d8SscAfTMph+2aW416TXB1/UZw="}},{"mechanisms":[],"roles":[{"role":"readWrite","<SENSITIVE>":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions":[],"scramSha
256Creds":{"iterationCount":15000,"salt":"GOkTsWgdrct5KSSQtTHC20myEJM76v5OMEGXOA==","serverKey":"ksqF9YIWnI50+bQJhjl0/zA1a0H0UpcNnzxnFEjciV4=","storedKey":"GyxjpwCp9hTHsK
5CX2ObkIs73NP2zL1VrwbCQTLDvGE="},"scramSha1Creds":{"iterationCount":10000,"salt":"4RAcRyAxnRCQQhcHWRDA2w==","serverKey":"59KQ8PQV/rS4zxSuVca/tQbDNWw=","storedKey":"v68O4b
x8u7/RNIks1WBvmXIJ+H8="}},{"mechanisms":[],"roles":[{"role":"readWrite","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions":[],"scramSha256Creds
":{"iterationCount":15000,"salt":"vYX0jOTF0NvPPmpQm1oz/b7v1/sAnFOMWdm5Pg==","serverKey":"MWuSeUedUk33YD57g/pVw+kV89vQK8OmTibRLl2hR0U=","storedKey":"ffXuxQ5HTf0FcH2FdcNvKi
gWSO/TgPdF0elXk9iYX3E="},"scramSha1Creds":{"iterationCount":10000,"salt":"zV90H0Z2XJ8sCiupCSK3PQ==","serverKey":"IwdxN4BVrGSqyLPXDhrbZKFsbtc=","storedKey":"e5XxJTwdueUyUy
Jd3ioqQFuEKbc="}},{"mechanisms":[],"roles":[{"role":"readWrite","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions
":[],"scramSha256Creds":{"iterationCount":15000,"salt":"0HwrZJa9FIMCy5r4erCq7o2gb/RSaHofCV1XMw==","serverKey":"4FTg/fstci+8W6BZE4jfyXpLJr9/f4zsuDiKrLnBcgg=","storedKey":"
lThVu1E2tv14Q7H58DYYNK1jlqXaIZDCp/Omp44wR1A="},"scramSha1Creds":{"iterationCount":10000,"salt":"gDxOqOLC16/e/WvhWSGDdA==","serverKey":"q6SKd30cOY+PFQnqRFMpdgmNTFA=","stor
edKey":"9swpmpNkjLofRRRprZWGrCBoolk="}},{"mechanisms":[],"roles":[{"role":"dbAdmin","db":"<SENSITIVE>"},{"role":"userAdminAnyDatabase","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"},{"role":"readWrite","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authenticationRestrictions":[],"scramSha256Creds":{"iterationC
ount":15000,"salt":"m40BqXb1jbVT4NIN9NjAdbTdQp84O6KtEbRjgA==","serverKey":"hztnbCDXJs0zBcwtaJsLquRtEgCHDykKj04SaQ3eLn8=","storedKey":"cftguOpTPNr4QYvL2XtrRlybPzi96CgzGoZ9
1EVZg2g="},"scramSha1Creds":{"iterationCount":10000,"salt":"SofiWm+P4s3RwvvIxflOOQ==","serverKey":"73knk0VrQPm6PWSYM5PFYwUK1lA=","storedKey":"wnYGbRIv2qPtcpv3j4r+lUX8x/4=
"}},{"mechanisms":[],"roles":[{"role":"read","db":"<SENSITIVE>"},{"role":"read","db":"<SENSITIVE>"},{"role":"changestreamrole","db":"<SENSITIVE>"}],"user":"<SENSITIVE>","db":"<SENSITIVE>","authentica
tionRestrictions":[],"scramSha256Creds":{"iterationCount":15000,"salt":"wzzV+4JFqwIGw1mAzRucb1oiIcYYR/gdcwZyJw==","serverKey":"Ty105G/oxXhrv9UwgIqqXHO7ZxM5LYW9T/mta7uiQYo
=","storedKey":"63E/7kQy2g4O/MUd0a62q8pQNBtITkJ74dUagrRESO4="},"scramSha1Creds":{"iterationCount":10000,"salt":"R/PWGnO94tygxdNjIavdkQ==","serverKey":"Gr8vtcLPyb/pR/h8GRi
vOrY6/sE=","storedKey":"kwWqu/dnDzNTDOEu03TcgeDWxPY="}}],"disabled":false,"authoritativeSet":false,"autoAuthMechanisms":["SCRAM-SHA-256"],"autoAuthMechanism":"SCRAM-SHA-2
56","deploymentAuthMechanisms":["SCRAM-SHA-256"],"autoUser":"mms-automation","key":"8tQDoV1eZdKJvpc7cA8rtu939Glj1IgsL9CNE1nf7SuZMFw8Te47PmhA9Z1NPi27cRw5+bs16kenEAPP82V7v5
1Xcv5Q9xPZKUxltKlc3t9cfq2Q7Il42DJsrjhQUhne5lKNghWLRHPSFVb8IHbuImgPcvu7mPz6VsYClu6Lno5ewW3ziIvilIW/2xvpxqG0qz4jvz5/cmtTWeNn7VJzNOYwYurWdFfvdUDL+Z+kQqcbsa95SSYA8217h6aKE2gu
OwlVpK0VZBYCPACg+ID1dARawAHG7xCA92lFttymLfgu8kbUXeW6RxBsgwz5iuOXjiIrm8XpWhpWHLJNplf5YaGsqBMIbRlH3tAXGv6auqLaiGup3+kQXDJNwC7Juaa5F0FGXg+PdQPMOH4xv2SZy0zGHh988CaEtXhBVWiQ06
FhnNWyxziLCl8BGJpCbD2bsGiiUBcUHvxkCybARhguLYdnS60+tlJcMIr3rpt7MTgRuHhwki0gX1KcVmEe+tPeg57RdqcVcEEqqHYwc4Ghkk/PF/10BlsO0NiUZmJxZqow7ffSRZHtZ/VKW2og6CZp2V3BaYZmzYwHn5XFFRCD
NUu8mbwvtySQVSlVVY4GbKRkgepYsWrYGc20yPH7Hzni9b8N0zCmX8HPy5icn8+jf4z7BRw=","keyfile":"/var/lib/mongodb-mms-automation/authentication/keyfile","keyfileWindows":"%SystemDriv
e%\\MMSAutomation\\versions\\keyfile","autoPwd":"eX-rXNR2PB_ytwagyylk"},"tls":{"CAFilePath":"","clientCertificateMode":"OPTIONAL"},"mongoDbVersions":[{"name":"6.0.4","bui
lds":[{"platform":"linux","url":"","gitVersion":"","architecture":"amd64","flavor":"rhel","minOsVersion":"","maxOsVersion":"","modules":[]},{"platform":"linux","url":"","
gitVersion":"","architecture":"amd64","flavor":"ubuntu","minOsVersion":"","maxOsVersion":"","modules":[]}]}],"backupVersions":[],"monitoringVersions":[],"options":{"downl
oadBase":"/var/lib/mongodb-mms-automation"}}
```
- The agent health status of the faulty members:
- `kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json`
```
{"statuses":{"mongodb-0":{"IsInGoalState":true,"LastMongoUpTime":1725441832,"ExpectedToBeUp":true,"ReplicationStatus":1}},"mmsStatus":{"mongodb-0":{"name":"mongodb-0","la
stGoalVersionAchieved":6,"plans":[{"automationConfigVersion":6,"started":"2024-09-02T08:56:39.289177326Z","completed":"2024-09-02T08:56:44.316657181Z","moves":[{"move":"S
tart","moveDoc":"Start the process","steps":[{"step":"StartFresh","stepDoc":"Start a mongo instance (start fresh)","isWaitStep":false,"started":"2024-09-02T08:56:39.2891
92036Z","completed":"2024-09-02T08:56:43.2738461Z","result":"success"}]},{"move":"WaitAllRsMembersUp","moveDoc":"Wait until all members of this process' repl set are up",
"steps":[{"step":"WaitAllRsMembersUp","stepDoc":"Wait until all members of this process' repl set are up","isWaitStep":true,"started":"2024-09-02T08:56:43.273886773Z","co
mpleted":"2024-09-02T08:56:43.274018861Z","result":"success"}]},{"move":"RsInit","moveDoc":"Initialize a replica set including the current MongoDB process","steps":[{"ste
p":"RsInit","stepDoc":"Initialize a replica set","isWaitStep":false,"started":"2024-09-02T08:56:43.274046073Z","completed":null,"result":"error"}]},{"move":"WaitFeatureCo
mpatibilityVersionCorrect","moveDoc":"Wait for featureCompatibilityVersion to be right","steps":[{"step":"WaitFeatureCompatibilityVersionCorrect","stepDoc":"Wait for feat
ureCompatibilityVersion to be right","isWaitStep":true,"started":null,"completed":null,"result":""}]}]},{"automationConfigVersion":6,"started":"2024-09-03T12:29:37.736565
486Z","completed":"2024-09-03T12:29:42.80790282Z","moves":[{"move":"EnsureAutomationCredentials","moveDoc":"Ensure the automation user exists","steps":[{"step":"EnsureAut
omationCredentials","stepDoc":"Ensure the automation user exists","isWaitStep":false,"started":"2024-09-03T12:29:37.736594109Z","completed":"2024-09-03T12:29:42.806979734
Z","result":"success"}]}]}],"errorCode":0,"errorString":""}}}
```
- The verbose agent logs of the faulty members:
- `kubectl exec -it mongo-0 -c mongodb-agent -- cat /var/log/mongodb-mms-automation/automation-agent-verbose.log`
```
[2024-09-03T12:57:47.152+0000] [.debug] [src/mongoctl/processctl.go:ProcessType:1306] <mongodb-0> [12:57:47.152] Checking process type of mongodb-0.mongodb-svc.mongodb-su
rplus.svc.cluster.local:27017 (local=false)...
[2024-09-03T12:57:47.152+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.152] Starting RunCommand(dbName=admin, cmd=[{isMaste
r 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.153] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{isMa
ster 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":27},"signature":{"
hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"connectionId":13,"electionId":"7fffffff000000000000008d","hosts":["mongodb-0.mong
odb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","mongodb-1.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017"],"ismaster":true,"lastWrite":{"lastWriteDate":"2024-09-03
T12:57:47Z","majorityOpTime":{"t":141,"ts":{"T":1725368267,"I":27}},"majorityWriteDate":"2024-09-03T12:57:47Z","opTime":{"t":141,"ts":{"T":1725368267,"I":27}}},"localTime
":"2024-09-03T12:57:47.153Z","logicalSessionTimeoutMinutes":30,"maxBsonObjectSize":16777216,"maxMessageSizeBytes":48000000,"maxWireVersion":17,"maxWriteBatchSize":100000,
"me":"mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","minWireVersion":0,"ok":1,"operationTime":{"T":1725368267,"I":27},"primary":"mongodb-0.mongodb-svc.mo
ngodb-<SENSITIVE>.svc.cluster.local:27017","readOnly":false,"secondary":false,"setName":"mongodb","setVersion":1,"topologyVersion":{"counter":10,"processId":"66d57dc915c38471
0a9d334d"}}
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.153] Starting RunCommand(dbName=admin, cmd=[{getPara
meter 1} {featureCompatibilityVersion 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.153] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getP
arameter 1} {featureCompatibilityVersion 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"
T":1725368267,"I":27},"signature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"featureCompatibilityVersion":{"version":"6.0"
},"ok":1,"operationTime":{"T":1725368267,"I":27}}
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:FeatureCompatibilityVersion:3796] <mongodb-0> [12:57:47.153] FeatureCompatibilityVersion on 3.6+ is 6.
0
[2024-09-03T12:57:47.153+0000] [.debug] [src/mongoctl/processctl.go:ParsedArgs:2414] <mongodb-0> [12:57:47.153] Getting parsed args
[2024-09-03T12:57:47.154+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.154] Starting RunCommand(dbName=admin, cmd=[{isMaste
r 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.154+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.154] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{isMa
ster 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":27},"signature":{"
hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"connectionId":13,"electionId":"7fffffff000000000000008d","hosts":["mongodb-0.mong
odb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","mongodb-1.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017"],"ismaster":true,"lastWrite":{"lastWriteDate":"2024-09-03
T12:57:47Z","majorityOpTime":{"t":141,"ts":{"T":1725368267,"I":27}},"majorityWriteDate":"2024-09-03T12:57:47Z","opTime":{"t":141,"ts":{"T":1725368267,"I":27}}},"localTime
":"2024-09-03T12:57:47.154Z","logicalSessionTimeoutMinutes":30,"maxBsonObjectSize":16777216,"maxMessageSizeBytes":48000000,"maxWireVersion":17,"maxWriteBatchSize":100000,
"me":"mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","minWireVersion":0,"ok":1,"operationTime":{"T":1725368267,"I":27},"primary":"mongodb-0.mongodb-svc.mo
ngodb-<SENSITIVE>.svc.cluster.local:27017","readOnly":false,"secondary":false,"setName":"mongodb","setVersion":1,"topologyVersion":{"counter":10,"processId":"66d57dc915c38471
0a9d334d"}}
[2024-09-03T12:57:47.154+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.154] Starting RunCommand(dbName=admin, cmd=[{getCmdL
ineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.155] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getC
mdLineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":27},"signatu
re":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"argv":["mongod","-f","/data/automation-mongod.conf"],"ok":1,"operationTime"
:{"T":1725368267,"I":27},"parsed":{"config":"/data/automation-mongod.conf","net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"au
thorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath"
:"/data","engine":"wiredTiger","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}}
[2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:overwriteWithRedactedArgsFromConfig:2495] <mongodb-0> [12:57:47.155] Attempting to overwrite redacted
args with real value of redacted args from config file [/data/automation-mongod.conf] for host [27017]
[2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:overwriteWithRedactedArgsFromConfig:2506] <mongodb-0> [12:57:47.155] Overwrote redacted args with real
value of redacted args from config file [/data/automation-mongod.conf] for host [27017]
[2024-09-03T12:57:47.155+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.155] Starting RunCommand(dbName=admin, cmd=[{getPara
meter 1} {tlsMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.156+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.156] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getP
arameter 1} {tlsMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":28
},"signature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"ok":1,"operationTime":{"T":1725368267,"I":28},"tlsMode":"disabled
"}
[2024-09-03T12:57:47.156+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.156] Starting RunCommand(dbName=admin, cmd=[{getPara
meter 1} {clusterAuthMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.156+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.156] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getP
arameter 1} {clusterAuthMode 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":172536826
7,"I":28},"signature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"clusterAuthMode":"keyFile","ok":1,"operationTime":{"T":17
25368267,"I":28}}
[2024-09-03T12:57:47.157+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.157] Starting RunCommand(dbName=admin, cmd=[{getDefa
ultRWConcern 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.157+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.157] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getD
efaultRWConcern 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":28},"si
gnature":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"defaultReadConcern":{"level":"local"},"defaultReadConcernSource":"impl
icit","defaultWriteConcern":{"w":"majority","wtimeout":0},"defaultWriteConcernSource":"implicit","localUpdateWallClockTime":"2024-09-03T12:57:47.157Z","ok":1,"operationTi
me":{"T":1725368267,"I":28}}
[2024-09-03T12:57:47.157+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.157] Starting RunCommand(dbName=admin, cmd=[{getCmdL
ineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.158+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.158] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{getC
mdLineOpts 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"$clusterTime":{"clusterTime":{"T":1725368267,"I":28},"signatu
re":{"hash":{"Subtype":0,"Data":"hRq/77/Au3zw50gEi8bDz6yrW6o="},"keyId":7356884261963563010}},"argv":["mongod","-f","/data/automation-mongod.conf"],"ok":1,"operationTime"
:{"T":1725368267,"I":28},"parsed":{"config":"/data/automation-mongod.conf","net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"mongodb"},"security":{"au
thorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath"
:"/data","engine":"wiredTiger","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}}
[2024-09-03T12:57:47.159+0000] [.info] [src/mongoclientservice/mongoclientservice.go:func1:1619] [12:57:47.159] Testing auth with username __system db=local to mongodb-0.
mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) connectMode=SingleConnect ipversion=0 tls=false
[2024-09-03T12:57:47.169+0000] [.info] [src/mongoctl/processctl.go:GetKeyHashes:2080] <mongodb-0> [12:57:47.169] Able to successfully auth to mongodb-0.mongodb-svc.mongod
b-<SENSITIVE>.svc.cluster.local:27017 (local=false) using desired auth key
[2024-09-03T12:57:47.169+0000] [.debug] [state/stateutil/fickleutil.go:CurrentFickle:51] <mongodb-0> [12:57:47.169] Starting CurrentFickle...
[2024-09-03T12:57:47.169+0000] [.debug] [src/mongoctl/replsetctl.go:ReplSetStatus:285] <mongodb-0> [12:57:47.169] Getting repl set status for process mongodb-0.mongodb-sv
c.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false)...
[2024-09-03T12:57:47.169+0000] [.debug] [src/mongoctl/processctl.go:RunCommandWithTimeout:1145] <mongodb-0> [12:57:47.169] Starting RunCommand(dbName=admin, cmd=[{replSet
GetStatus 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) ...
[2024-09-03T12:57:47.170+0000] [.debug] [src/mongoctl/processctl.go:func1:1168] <mongodb-0> [12:57:47.170] ...Finished with RunCommandWithTimeout(dbName=admin, cmd=[{repl
SetGetStatus 1}]) to mongodb-0.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017 (local=false) with result={"members":[{"_id":0,"name":"mongodb-0.mongodb-svc.mongodb-su
rplus.svc.cluster.local:27017","health":1,"state":1,"self":true,"optime":{"ts":{"T":1725368267,"I":29},"t":141},"optimedate":"2024-09-03T12:57:47Z"},{"_id":1,"name":"mong
odb-1.mongodb-svc.mongodb-<SENSITIVE>.svc.cluster.local:27017","health":1,"state":2,"self":false,"optime":{"ts":{"T":1725368266,"I":67},"t":141},"optimedate":"2024-09-03T12:5
7:46Z"}],"errmsg":"","myState":1,"optimes":{"lastCommittedOpTime":{"ts":{"T":1725368267,"I":28},"t":141},"appliedOpTime":{"ts":{"T":1725368267,"I":29},"t":141},"durableOp
Time":{"ts":{"T":1725368267,"I":29},"t":141}}}
```
tried adding arbiter to PS replicaset to make it PSA. 2 members are fine but after adding 1 arbiter, operator goes into some crashloop with agent goal state. I checked go code for that part yet did not udnderstand this behaviour