The Ops Manager is responsible for facilitating workloads such as backing up data, monitoring database performance and more. To make your multi-cluster Ops Manager and the Application Database deployment resilient to entire data center or zone failures, deploy the Ops Manager Application and the Application Database on multiple Kubernetes clusters.
Prerequisites
Before you begin the following procedure, perform the following actions:
Install
kubectl.Complete the GKE Clusters procedure or the equivalent.
Complete the TLS Certificates procedure or the equivalent.
Complete the ExternalDNS procedure or the equivalent.
Complete the Deploy the MongoDB Operator procedure.
Set the required environment variables as follows:
# This script builds on top of the environment configured in the setup guides.# It depends (uses) the following env variables defined there to work correctly.# If you don't use the setup guide to bootstrap the environment, then define them here.# ${K8S_CLUSTER_0_CONTEXT_NAME}# ${K8S_CLUSTER_1_CONTEXT_NAME}# ${K8S_CLUSTER_2_CONTEXT_NAME}# ${OM_NAMESPACE}# ${CUSTOM_DOMAIN}# ${DNS_ZONE}
export S3_OPLOG_BUCKET_NAME=s3-oplog-storeexport S3_SNAPSHOT_BUCKET_NAME=s3-snapshot-store
# If you use your own S3 storage - set the values accordingly.# By default we install Minio to handle S3 storage and here are set the default credentials.export S3_ENDPOINT="minio.tenant-tiny.svc.cluster.local"export S3_ACCESS_KEY="console"export S3_SECRET_KEY="console123"
export OPS_MANAGER_VERSION="8.0.5"export APPDB_VERSION="8.0.5-ent"
export OPS_MANAGER_EXTERNAL_DOMAIN="opsmanager.${CUSTOM_DOMAIN}"export APPDB_CLUSTER_0_EXTERNAL_DOMAIN="${K8S_CLUSTER_0}.${CUSTOM_DOMAIN}"export APPDB_CLUSTER_1_EXTERNAL_DOMAIN="${K8S_CLUSTER_1}.${CUSTOM_DOMAIN}"export APPDB_CLUSTER_2_EXTERNAL_DOMAIN="${K8S_CLUSTER_2}.${CUSTOM_DOMAIN}"Source Code
You can find all included source code in the MongoDB Kubernetes Operator repository.
Procedure
Generate TLS certificates.
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" apply -f - <<EOFapiVersion: cert-manager.io/v1kind: Certificatemetadata: name: om-certspec: dnsNames: - ${OPS_MANAGER_EXTERNAL_DOMAIN} duration: 240h0m0s issuerRef: name: my-ca-issuer kind: ClusterIssuer renewBefore: 120h0m0s secretName: cert-prefix-om-cert usages: - server auth - client auth---apiVersion: cert-manager.io/v1kind: Certificatemetadata: name: om-db-certspec: dnsNames: - "*.${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" - "*.${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" - "*.${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" duration: 240h0m0s issuerRef: name: my-ca-issuer kind: ClusterIssuer renewBefore: 120h0m0s secretName: cert-prefix-om-db-cert usages: - server auth - client authEOFAdd TLS certificate to GCP.
mkdir -p certs
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get secret cert-prefix-om-cert -o jsonpath="{.data['tls\.crt']}" | base64 --decode > certs/tls.crtkubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get secret cert-prefix-om-cert -o jsonpath="{.data['tls\.key']}" | base64 --decode > certs/tls.key
gcloud compute ssl-certificates create om-certificate --certificate=certs/tls.crt --private-key=certs/tls.keyCreate the required Kubernetes components for a load balancer.
This load balancer distributes traffic between all the replicas of Ops Manager across all 3 clusters.
gcloud compute firewall-rules create fw-ops-manager-hc \ --action=allow \ --direction=ingress \ --target-tags=mongodb \ --source-ranges=130.211.0.0/22,35.191.0.0/16 \ --rules=tcp:8443
gcloud compute health-checks create https om-healthcheck \ --use-serving-port \ --request-path=/monitor/health
gcloud compute backend-services create om-backend-service \ --protocol HTTPS \ --health-checks om-healthcheck \ --global
gcloud compute url-maps create om-url-map \ --default-service om-backend-service
gcloud compute target-https-proxies create om-lb-proxy \ --url-map om-url-map \ --ssl-certificates=om-certificate
gcloud compute forwarding-rules create om-forwarding-rule \ --global \ --target-https-proxy=om-lb-proxy \ --ports=443NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLEDfw-ops-manager-hc default INGRESS 1000 tcp:8443 FalseNAME PROTOCOLom-healthcheck HTTPSNAME BACKENDS PROTOCOLom-backend-service HTTPSNAME DEFAULT_SERVICEom-url-map backendServices/om-backend-serviceNAME SSL_CERTIFICATES URL_MAP REGION CERTIFICATE_MAPom-lb-proxy om-certificate om-url-mapAdd an "A" record to your DNS zone with your external domain.
ip_address=$(gcloud compute forwarding-rules describe om-forwarding-rule --global --format="get(IPAddress)")
gcloud dns record-sets create "${OPS_MANAGER_EXTERNAL_DOMAIN}" --zone="${DNS_ZONE}" --type="A" --ttl="300" --rrdatas="${ip_address}"Create credentials for the Ops Manager admin user.
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" --namespace "${OM_NAMESPACE}" create secret generic om-admin-user-credentials \ --from-literal=Username="admin" \ --from-literal=Password="Passw0rd@" \ --from-literal=FirstName="Jane" \ --from-literal=LastName="Doe"Deploy Ops Manager.
kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOFapiVersion: mongodb.com/v1kind: MongoDBOpsManagermetadata: name: omspec: topology: MultiCluster version: "${OPS_MANAGER_VERSION}" adminCredentials: om-admin-user-credentials externalConnectivity: type: ClusterIP annotations: cloud.google.com/neg: '{"exposed_ports": {"8443":{}}}' opsManagerURL: "https://${OPS_MANAGER_EXTERNAL_DOMAIN}" security: certsSecretPrefix: cert-prefix tls: ca: ca-issuer clusterSpecList: - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" members: 1 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" members: 2 applicationDatabase: version: "${APPDB_VERSION}" topology: MultiCluster security: certsSecretPrefix: cert-prefix tls: ca: ca-issuer clusterSpecList: - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" members: 2 externalAccess: externalDomain: "${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" members: 2 externalAccess: externalDomain: "${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}" members: 1 externalAccess: externalDomain: "${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" backup: enabled: falseEOFWait for the Kubernetes Operator to enter a pending state.
Wait for both the Application Database and Ops Manager deployments to complete.
echo "Waiting for Application Database to reach Pending phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s
echo "Waiting for Ops Manager to reach Pending phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Pending opsmanager/om --timeout=600sWaiting for Application Database to reach Pending phase...mongodbopsmanager.mongodb.com/om condition metWaiting for Ops Manager to reach Pending phase...mongodbopsmanager.mongodb.com/om condition metSet up load balancer services.
svcneg0=$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg -o=jsonpath='{.items[0].metadata.name}')
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" \ wait svcneg "${svcneg0}" --for=condition=Initialized --timeout=300s \ || { kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg "${svcneg0}" -o yaml; exit 1; }
gcloud compute backend-services add-backend om-backend-service \ --global \ --network-endpoint-group="${svcneg0}" \ --network-endpoint-group-zone="${K8S_CLUSTER_0_ZONE}" \ --balancing-mode RATE --max-rate-per-endpoint 5svcneg1=$(kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg -o=jsonpath='{.items[0].metadata.name}')
kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" \ wait svcneg "${svcneg1}" --for=condition=Initialized --timeout=300s \ || { kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg "${svcneg1}" -o yaml; exit 1; }
gcloud compute backend-services add-backend om-backend-service \ --global \ --network-endpoint-group="${svcneg1}" \ --network-endpoint-group-zone="${K8S_CLUSTER_1_ZONE}" \ --balancing-mode RATE --max-rate-per-endpoint 5Wait for Ops Manager to enter a running state.
echo "Waiting for Application Database to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900secho; echo "Waiting for Ops Manager to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900secho; echo "MongoDBOpsManager resource"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/omecho; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsecho; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsWaiting for Application Database to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
Waiting for Ops Manager to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
MongoDBOpsManager resourceNAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGSom 8.0.5 Running Running Disabled 17m
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-mongodb-kubx7d456e3-8794NAME READY STATUS RESTARTS AGEom-0-0 1/1 Running 0 15mom-db-0-0 3/3 Running 0 3m51som-db-0-1 3/3 Running 0 4m53s
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-mongodb-kubx7d456e3-8794NAME READY STATUS RESTARTS AGEom-1-0 1/1 Running 0 15mom-1-1 1/1 Running 0 10mom-db-1-0 3/3 Running 0 2m3som-db-1-1 3/3 Running 0 2m56sInstall Minio.
kubectl kustomize "github.com/minio/operator/resources/?timeout=120&ref=v5.0.12" | \ kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
kubectl kustomize "github.com/minio/operator/examples/kustomization/tenant-tiny?timeout=120&ref=v5.0.12" | \ kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
# add two buckets to the tenant configkubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "tenant-tiny" patch tenant/myminio \ --type='json' \ -p="[{\"op\": \"add\", \"path\": \"/spec/buckets\", \"value\": [{\"name\": \"${S3_OPLOG_BUCKET_NAME}\"}, {\"name\": \"${S3_SNAPSHOT_BUCKET_NAME}\"}]}]"Configure Kubernetes Secrets for Ops Manager backups.
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-access-secret \ --from-literal=accessKey="${S3_ACCESS_KEY}" \ --from-literal=secretKey="${S3_SECRET_KEY}"
# minio TLS secrets are signed with the default k8s root CAkubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-ca-cert \ --from-literal=ca.crt="$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n kube-system get configmap kube-root-ca.crt -o jsonpath="{.data.ca\.crt}")"Enable S3 (Minio) backups in Ops Manager.
kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOFapiVersion: mongodb.com/v1kind: MongoDBOpsManagermetadata: name: omspec: topology: MultiCluster version: "${OPS_MANAGER_VERSION}" adminCredentials: om-admin-user-credentials externalConnectivity: type: ClusterIP annotations: cloud.google.com/neg: '{"exposed_ports": {"8443":{}}}' opsManagerURL: "https://${OPS_MANAGER_EXTERNAL_DOMAIN}" security: certsSecretPrefix: cert-prefix tls: ca: ca-issuer clusterSpecList: - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" members: 1 backup: members: 0 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" members: 2 backup: members: 0 - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}" members: 0 backup: members: 1 applicationDatabase: version: "${APPDB_VERSION}" topology: MultiCluster security: certsSecretPrefix: cert-prefix tls: ca: ca-issuer clusterSpecList: - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" members: 2 externalAccess: externalDomain: "${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" members: 2 externalAccess: externalDomain: "${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}" members: 1 externalAccess: externalDomain: "${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" backup: enabled: true s3Stores: - name: my-s3-block-store s3SecretRef: name: "s3-access-secret" pathStyleAccessEnabled: true s3BucketEndpoint: "${S3_ENDPOINT}" s3BucketName: "${S3_SNAPSHOT_BUCKET_NAME}" customCertificateSecretRefs: - name: s3-ca-cert key: ca.crt s3OpLogStores: - name: my-s3-oplog-store s3SecretRef: name: "s3-access-secret" s3BucketEndpoint: "${S3_ENDPOINT}" s3BucketName: "${S3_OPLOG_BUCKET_NAME}" pathStyleAccessEnabled: true customCertificateSecretRefs: - name: s3-ca-cert key: ca.crtEOFWait for Ops Manager to enter a running state.
echo; echo "Waiting for Backup to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.backup.phase}'=Running opsmanager/om --timeout=1200secho "Waiting for Application Database to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600secho; echo "Waiting for Ops Manager to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600secho; echo "MongoDBOpsManager resource"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/omecho; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsecho; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsecho; echo "Pods running in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsWaiting for Backup to reach Running phase...mongodbopsmanager.mongodb.com/om condition metWaiting for Application Database to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
Waiting for Ops Manager to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
MongoDBOpsManager resourceNAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGSom 8.0.5 Running Running Running 21m
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-mongodb-kubx7d456e3-8794NAME READY STATUS RESTARTS AGEom-0-0 1/1 Running 0 19mom-db-0-0 3/3 Running 0 7m48som-db-0-1 3/3 Running 0 8m50s
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-mongodb-kubx7d456e3-8794NAME READY STATUS RESTARTS AGEom-1-0 1/1 Running 0 19mom-1-1 1/1 Running 0 14mom-db-1-0 3/3 Running 0 6m1som-db-1-1 3/3 Running 0 6m54s
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2-mongodb-kubx7d456e3-8794NAME READY STATUS RESTARTS AGEom-2-backup-daemon-0 1/1 Running 0 3m27som-db-2-0 3/3 Running 0 4m56sCreate MongoDB Organization and get credentials.
To configure credentials, you must create an Ops Manager organization, generate programmatic API keys in the Ops Manager UI, and create a secret with your Load Balancer IP. See Create Credentials for the Kubernetes Operator to learn more.