The Ops Manager is responsible for facilitating workloads such as backing up data, monitoring database performance and more. To make your multi-cluster Ops Manager and the Application Database deployment resilient to entire data center or zone failures, deploy the Ops Manager Application and the Application Database on multiple Kubernetes clusters.
Prerequisites
Before you begin the following procedure, perform the following actions:
Install
kubectl.Complete the GKE Clusters procedure or the equivalent.
Complete the TLS Certificates procedure or the equivalent.
Complete the Istio Service mesh procedure or the equivalent.
Complete the Deploy the MongoDB Operator procedure.
Set the required environment variables as follows:
# This script builds on top of the environment configured in the setup guides. # It depends (uses) the following env variables defined there to work correctly. # If you don't use the setup guide to bootstrap the environment, then define them here. # ${K8S_CLUSTER_0_CONTEXT_NAME} # ${K8S_CLUSTER_1_CONTEXT_NAME} # ${K8S_CLUSTER_2_CONTEXT_NAME} # ${OM_NAMESPACE} export S3_OPLOG_BUCKET_NAME=s3-oplog-store export S3_SNAPSHOT_BUCKET_NAME=s3-snapshot-store # If you use your own S3 storage - set the values accordingly. # By default we install Minio to handle S3 storage and here are set the default credentials. export S3_ENDPOINT="minio.tenant-tiny.svc.cluster.local" export S3_ACCESS_KEY="console" export S3_SECRET_KEY="console123" # (Optional) Change the following setting when using the external URL. # This env variable is used in OpenSSL configuration to generate # server certificates for Ops Manager Application. export OPS_MANAGER_EXTERNAL_DOMAIN="om-svc.${OM_NAMESPACE}.svc.cluster.local" export OPS_MANAGER_VERSION="8.0.4" export APPDB_VERSION="8.0.5"
Source Code
You can find all included source code in the MongoDB Kubernetes Operator repository.
Procedure
Generate TLS certificates.
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" apply -f - <<EOF apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: om-cert spec: dnsNames: - ${OPS_MANAGER_EXTERNAL_DOMAIN} - om-svc.${OM_NAMESPACE}.svc.cluster.local duration: 240h0m0s issuerRef: name: my-ca-issuer kind: ClusterIssuer renewBefore: 120h0m0s secretName: cert-prefix-om-cert usages: - server auth - client auth --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: om-db-cert spec: dnsNames: - "*.${OM_NAMESPACE}.svc.cluster.local" - "*.om-db-svc.${OM_NAMESPACE}.svc.cluster.local" duration: 240h0m0s issuerRef: name: my-ca-issuer kind: ClusterIssuer renewBefore: 120h0m0s secretName: cert-prefix-om-db-cert usages: - server auth - client auth EOF
Install Ops Manager.
At this point, you have prepared the environment and the Kubernetes Operator to deploy the Ops Manager resource.
Create the necessary credentials for the Ops Manager admin user that the Kubernetes Operator will create after deploying the Ops Manager Application instance:
1 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" --namespace "${OM_NAMESPACE}" create secret generic om-admin-user-credentials \ 2 --from-literal=Username="admin" \ 3 --from-literal=Password="Passw0rd@" \ 4 --from-literal=FirstName="Jane" \ 5 --from-literal=LastName="Doe" Deploy the simplest
MongoDBOpsManagercustom resource possible (with TLS enabled) on a single member cluster, which is also known as the operator cluster.This deployment is almost the same as the deployment for the single-cluster mode, but with
spec.topologyandspec.applicationDatabase.topologyset toMultiCluster.Deploying this way shows that a single Kubernetes cluster deployment is a special case of a multi-Kubernetes cluster deployment on a single Kubernetes member cluster. You can start deploying the Ops Manager Application and the Application Database on as many Kubernetes clusters as necessary from the beginning, and don't have to start with the deployment with only a single member Kubernetes cluster.
At this point, you have prepared the Ops Manager deployment to span more than one Kubernetes cluster, which you will do later in this procedure.
1 kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOF 2 apiVersion: mongodb.com/v1 3 kind: MongoDBOpsManager 4 metadata: 5 name: om 6 spec: 7 topology: MultiCluster 8 version: "${OPS_MANAGER_VERSION}" 9 adminCredentials: om-admin-user-credentials 10 externalConnectivity: 11 type: LoadBalancer 12 security: 13 certsSecretPrefix: cert-prefix 14 tls: 15 ca: ca-issuer 16 clusterSpecList: 17 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 18 members: 1 19 applicationDatabase: 20 version: "${APPDB_VERSION}" 21 topology: MultiCluster 22 security: 23 certsSecretPrefix: cert-prefix 24 tls: 25 ca: ca-issuer 26 clusterSpecList: 27 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 28 members: 3 29 backup: 30 enabled: false 31 EOF Wait for the Kubernetes Operator to pick up the work and reach the
status.applicationDatabase.phase=Pendingstate. Wait for both the Application Database and Ops Manager deployments to complete.1 echo "Waiting for Application Database to reach Pending phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s 1 Waiting for Application Database to reach Pending phase... 2 mongodbopsmanager.mongodb.com/om condition met Deploy Ops Manager. The Kubernetes Operator deploys Ops Manager by performing the following steps. It:
Deploys the Application Database's replica set nodes and waits for the MongoDB processes in the replica set to start running.
Deploys the Ops Manager Application instance with the Application Database's connection string and waits for it to become ready.
Adds the Monitoring MongoDB Agent containers to each Application Database's Pod.
Waits for both the Ops Manager Application and the Application Database Pods to start running.
1 echo "Waiting for Application Database to reach Running phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s 3 echo; echo "Waiting for Ops Manager to reach Running phase..." 4 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s 5 echo; echo "Waiting for Application Database to reach Pending phase (enabling monitoring)..." 6 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s 7 echo "Waiting for Application Database to reach Running phase..." 8 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s 9 echo; echo "Waiting for Ops Manager to reach Running phase..." 10 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s 11 echo; echo "MongoDBOpsManager resource" 12 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/om 13 echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}" 14 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 15 echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}" 16 kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 1 Waiting for Application Database to reach Running phase... 2 mongodbopsmanager.mongodb.com/om condition met 3 4 Waiting for Ops Manager to reach Running phase... 5 mongodbopsmanager.mongodb.com/om condition met 6 7 Waiting for Application Database to reach Pending phase (enabling monitoring)... 8 mongodbopsmanager.mongodb.com/om condition met 9 Waiting for Application Database to reach Running phase... 10 mongodbopsmanager.mongodb.com/om condition met 11 12 Waiting for Ops Manager to reach Running phase... 13 mongodbopsmanager.mongodb.com/om condition met 14 15 MongoDBOpsManager resource 16 NAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGS 17 om 8.0.4 Running Running Disabled 11m 18 19 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-67d0389d75b70a0007e5894a 20 NAME READY STATUS RESTARTS AGE 21 om-0-0 2/2 Running 0 7m35s 22 om-db-0-0 4/4 Running 0 37s 23 om-db-0-1 4/4 Running 0 115s 24 om-db-0-2 4/4 Running 0 2m57s 25 26 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-67d0389d75b70a0007e5894a Now that you have deployed a single-member cluster in a multi-cluster mode, you can reconfigure this deployment to span more than one Kubernetes cluster.
On the second member cluster, deploy two additional Application Database replica set members and one additional instance of the Ops Manager Application:
1 kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOF 2 apiVersion: mongodb.com/v1 3 kind: MongoDBOpsManager 4 metadata: 5 name: om 6 spec: 7 topology: MultiCluster 8 version: "${OPS_MANAGER_VERSION}" 9 adminCredentials: om-admin-user-credentials 10 externalConnectivity: 11 type: LoadBalancer 12 security: 13 certsSecretPrefix: cert-prefix 14 tls: 15 ca: ca-issuer 16 clusterSpecList: 17 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 18 members: 1 19 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" 20 members: 1 21 applicationDatabase: 22 version: "${APPDB_VERSION}" 23 topology: MultiCluster 24 security: 25 certsSecretPrefix: cert-prefix 26 tls: 27 ca: ca-issuer 28 clusterSpecList: 29 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 30 members: 3 31 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" 32 members: 2 33 backup: 34 enabled: false 35 EOF Wait for the Kubernetes Operator to pick up the work (pending phase):
1 echo "Waiting for Application Database to reach Pending phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s 1 Waiting for Application Database to reach Pending phase... 2 mongodbopsmanager.mongodb.com/om condition met Wait for the Kubernetes Operator to finish deploying all components:
1 echo "Waiting for Application Database to reach Running phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s 3 echo; echo "Waiting for Ops Manager to reach Running phase..." 4 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s 5 echo; echo "MongoDBOpsManager resource" 6 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/om 7 echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}" 8 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 9 echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}" 10 kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 1 Waiting for Application Database to reach Running phase... 2 mongodbopsmanager.mongodb.com/om condition met 3 4 Waiting for Ops Manager to reach Running phase... 5 mongodbopsmanager.mongodb.com/om condition met 6 7 MongoDBOpsManager resource 8 NAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGS 9 om 8.0.4 Running Running Disabled 19m 10 11 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-67d0389d75b70a0007e5894a 12 NAME READY STATUS RESTARTS AGE 13 om-0-0 2/2 Running 0 2m35s 14 om-db-0-0 4/4 Running 0 8m35s 15 om-db-0-1 4/4 Running 0 9m53s 16 om-db-0-2 4/4 Running 0 10m 17 18 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-67d0389d75b70a0007e5894a 19 NAME READY STATUS RESTARTS AGE 20 om-1-0 2/2 Running 0 3m5s 21 om-db-1-0 4/4 Running 0 7m49s 22 om-db-1-1 4/4 Running 0 5m54s
Enable backup.
In a multi-Kubernetes cluster deployment of the Ops Manager Application, you can configure
only S3-based backup storage. This procedure refers to S3_*
defined in env_variables.sh.
Optional. Install the MinIO Operator.
This procedure deploys S3-compatible storage for your backups using the MinIO Operator. You can skip this step if you have AWS S3 or other S3-compatible buckets available. Adjust the
S3_*variables accordingly in env_variables.sh in this case.1 kubectl kustomize "github.com/minio/operator/resources/?timeout=120&ref=v5.0.12" | \ 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f - 3 4 kubectl kustomize "github.com/minio/operator/examples/kustomization/tenant-tiny?timeout=120&ref=v5.0.12" | \ 5 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f - 6 7 # add two buckets to the tenant config 8 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "tenant-tiny" patch tenant/myminio \ 9 --type='json' \ 10 -p="[{\"op\": \"add\", \"path\": \"/spec/buckets\", \"value\": [{\"name\": \"${S3_OPLOG_BUCKET_NAME}\"}, {\"name\": \"${S3_SNAPSHOT_BUCKET_NAME}\"}]}]" 11 12 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "tenant-tiny" istio-injection=enabled --overwrite Before you configure and enable backup, create secrets:
s3-access-secret- contains S3 credentials.s3-ca-cert- contains a CA certificate that issued the bucket's server certificate. In the case of the sample MinIO deployment used in this procedure, the default Kubernetes Root CA certificate is used to sign the certificate. Because it's not a publicly trusted CA certificate, you must provide it so that Ops Manager can trust the connection.
If you use publicly trusted certificates, you may skip this step and remove the values from the
spec.backup.s3Stores.customCertificateSecretRefsandspec.backup.s3OpLogStores.customCertificateSecretRefssettings.1 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-access-secret \ 2 --from-literal=accessKey="${S3_ACCESS_KEY}" \ 3 --from-literal=secretKey="${S3_SECRET_KEY}" 4 5 # minio TLS secrets are signed with the default k8s root CA 6 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-ca-cert \ 7 --from-literal=ca.crt="$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n kube-system get configmap kube-root-ca.crt -o jsonpath="{.data.ca\.crt}")"
Re-deploy Ops Manager with backup enabled.
The Kubernetes Operator can configure and deploy all components, the Ops Manager Application, the Backup Daemon instances, and the Application Database's replica set nodes in any combination on any member clusters for which you configure the Kubernetes Operator.
To illustrate the flexibility of the multi-Kubernetes cluster deployment configuration, deploy only one Backup Daemon instance on the third member cluster and specify zero Backup Daemon members for the first and second clusters.
1 kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOF 2 apiVersion: mongodb.com/v1 3 kind: MongoDBOpsManager 4 metadata: 5 name: om 6 spec: 7 topology: MultiCluster 8 version: "${OPS_MANAGER_VERSION}" 9 adminCredentials: om-admin-user-credentials 10 externalConnectivity: 11 type: LoadBalancer 12 security: 13 certsSecretPrefix: cert-prefix 14 tls: 15 ca: ca-issuer 16 clusterSpecList: 17 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 18 members: 1 19 backup: 20 members: 0 21 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" 22 members: 1 23 backup: 24 members: 0 25 - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}" 26 members: 0 27 backup: 28 members: 1 29 configuration: # to avoid configuration wizard on first login 30 mms.adminEmailAddr: email@example.com 31 mms.fromEmailAddr: email@example.com 32 mms.ignoreInitialUiSetup: "true" 33 mms.mail.hostname: smtp@example.com 34 mms.mail.port: "465" 35 mms.mail.ssl: "true" 36 mms.mail.transport: smtp 37 mms.minimumTLSVersion: TLSv1.2 38 mms.replyToEmailAddr: email@example.com 39 applicationDatabase: 40 version: "${APPDB_VERSION}" 41 topology: MultiCluster 42 security: 43 certsSecretPrefix: cert-prefix 44 tls: 45 ca: ca-issuer 46 clusterSpecList: 47 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 48 members: 3 49 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" 50 members: 2 51 backup: 52 enabled: true 53 s3Stores: 54 - name: my-s3-block-store 55 s3SecretRef: 56 name: "s3-access-secret" 57 pathStyleAccessEnabled: true 58 s3BucketEndpoint: "${S3_ENDPOINT}" 59 s3BucketName: "${S3_SNAPSHOT_BUCKET_NAME}" 60 customCertificateSecretRefs: 61 - name: s3-ca-cert 62 key: ca.crt 63 s3OpLogStores: 64 - name: my-s3-oplog-store 65 s3SecretRef: 66 name: "s3-access-secret" 67 s3BucketEndpoint: "${S3_ENDPOINT}" 68 s3BucketName: "${S3_OPLOG_BUCKET_NAME}" 69 pathStyleAccessEnabled: true 70 customCertificateSecretRefs: 71 - name: s3-ca-cert 72 key: ca.crt 73 EOF Wait until the Kubernetes Operator finishes its configuration:
1 echo; echo "Waiting for Backup to reach Running phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.backup.phase}'=Running opsmanager/om --timeout=1200s 3 echo "Waiting for Application Database to reach Running phase..." 4 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s 5 echo; echo "Waiting for Ops Manager to reach Running phase..." 6 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s 7 echo; echo "MongoDBOpsManager resource" 8 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/om 9 echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}" 10 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 11 echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}" 12 kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 13 echo; echo "Pods running in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}" 14 kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 1 Waiting for Backup to reach Running phase... 2 mongodbopsmanager.mongodb.com/om condition met 3 Waiting for Application Database to reach Running phase... 4 mongodbopsmanager.mongodb.com/om condition met 5 6 Waiting for Ops Manager to reach Running phase... 7 mongodbopsmanager.mongodb.com/om condition met 8 9 MongoDBOpsManager resource 10 NAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGS 11 om 8.0.4 Running Running Running 24m 12 13 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-67d0389d75b70a0007e5894a 14 NAME READY STATUS RESTARTS AGE 15 om-0-0 2/2 Running 0 3m56s 16 om-db-0-0 4/4 Running 0 13m 17 om-db-0-1 4/4 Running 0 14m 18 om-db-0-2 4/4 Running 0 15m 19 20 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-67d0389d75b70a0007e5894a 21 NAME READY STATUS RESTARTS AGE 22 om-1-0 2/2 Running 0 3m55s 23 om-db-1-0 4/4 Running 0 12m 24 om-db-1-1 4/4 Running 0 10m 25 26 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2-67d0389d75b70a0007e5894a 27 NAME READY STATUS RESTARTS AGE 28 om-2-backup-daemon-0 2/2 Running 0 113s
Create credentials for the Kubernetes Operator.
To configure credentials, you must create an Ops Manager organization, generate programmatic API keys in the Ops Manager UI, and create a secret with your Load Balancer IP. See Create Credentials for the Kubernetes Operator to learn more.