Docs Menu
Docs Home
/
MongoDB Enterprise Kubernetes Operator
/

Deploy Ops Manager Resources on Multiple Kubernetes Clusters

On this page

  • Overview
  • Prerequisites
  • Procedure

To make your multi-cluster Ops Manager and the Application Database deployment resilient to entire data center or zone failures, deploy the Ops Manager Application and the Application Database on multiple Kubernetes clusters.

To learn more about the architecture, networking, limitations, and peformance of multi-Kubernetes cluster deployments for Ops Manager resources, see:

When you deploy the Ops Manager Application and the Application Database using the procedure in this section, you:

  1. Use GKE (Google Kubernetes Engine) and Istio service mesh as tools that help demonstrate the multi-Kubernetes cluster deployment.

  2. Install the Kubernetes Operator on one of the member Kubernetes clusters known as the operator cluster. The operator cluster acts as a Hub in the "Hub and Spoke" pattern used by the Kubernetes Operator to manage deployments on multiple Kubernetes clusters.

  3. Deploy the operator cluster in the $OPERATOR_NAMESPACE and configure this cluster to watch $NAMESPACE and manage all member Kubernetes clusters.

  4. Deploy the Application Database and the Ops Manager Application on a single member Kubernetes cluster to demonstrate similarity of a multi-cluster deployment to a single cluster deployment. A single cluster deployment with spec.topology and spec.applicationDatabase.topology set to MultiCluster prepares the deployment for adding more Kubernetes clusters to it.

  5. Deploy an additional Application Database replica set on the second member Kubernetes cluster to improve Application Database's resiliency. You also deploy an additional Ops Manager Application instance in the second member Kubernetes cluster.

  6. Create valid certificates for TLS encryption, and establish TLS-encrypted connections to and from the Ops Manager Application and between the Application Database's replica set members. When running over HTTPS, Ops Manager runs on port 8443 by default.

  7. Enable backup using S3-compatible storage and deploy the Backup Daemon on the third member Kubernetes cluster. To simplify setting up S3-compatible storage buckets, you deploy the MinIO Operator. You enable the Backup Daemon only on one member cluster in your deployment. However, you can configure other member clusters to host the Backup Daemon resources as well. Only S3 backups are supported in multi-cluster Ops Manager deployments.

Before you can begin the deployment, install the following required tools:

Install gcloud CLI and authorize into it:

gcloud auth login

The kubectl mongodb plugin automates the configuration of the Kubernetes clusters. This allows the Kubernetes Operator to deploy resources, necessary roles, and services for accounts for the Ops Manager Application, Application Database, and MongoDB resources on these clusters.

To install the kubectl mongodb plugin:

1

Download your desired Kubernetes Operator package version from the Release Page of the MongoDB Enterprise Kubernetes Operator Repository.

The package's name uses this pattern: kubectl-mongodb_{{ .Version }}_{{ .Os }}_{{ .Arch }}.tar.gz.

Use one of the following packages:

  • kubectl-mongodb_{{ .Version }}_darwin_amd64.tar.gz

  • kubectl-mongodb_{{ .Version }}_darwin_arm64.tar.gz

  • kubectl-mongodb_{{ .Version }}_linux_amd64.tar.gz

  • kubectl-mongodb_{{ .Version }}_linux_arm64.tar.gz

2

Unpack the package, as in the following example:

tar -zxvf kubectl-mongodb_<version>_darwin_amd64.tar.gz
3

Find the kubectl-mongodb binary in the unpacked directory and move it to its desired destination, inside the PATH for the Kubernetes Operator user, as shown in the following example:

mv kubectl-mongodb /usr/local/bin/kubectl-mongodb

Now you can run the kubectl mongodb plugin using the following commands:

kubectl mongodb multicluster setup
kubectl mongodb multicluster recover

To learn more about the supported flags, see the MongoDB kubectl plugin Reference.

Clone the MongoDB Enterprise Kubernetes Operator repository, change into the mongodb-enterprise-kubernetes directory, and check out the current version.

git clone https://github.com/mongodb/mongodb-enterprise-kubernetes.git
cd mongodb-enterprise-kubernetes
git checkout 1.26
cd public/samples/ops-manager-multi-cluster

Important

Some steps in this guide work only if you run them from the public/samples/ops-manager-multi-cluster directory.

All steps in this guide reference the environment variables defined in env_variables.sh.

1export MDB_GKE_PROJECT="### Set your GKE project name here ###"
2
3export NAMESPACE="mongodb"
4export OPERATOR_NAMESPACE="mongodb-operator"
5
6# comma-separated key=value pairs
7# export OPERATOR_ADDITIONAL_HELM_VALUES=""
8
9# Adjust the values for each Kubernetes cluster in your deployment.
10# The deployment script references the following variables to get values for each cluster.
11export K8S_CLUSTER_0="k8s-mdb-0"
12export K8S_CLUSTER_0_ZONE="europe-central2-a"
13export K8S_CLUSTER_0_NUMBER_OF_NODES=3
14export K8S_CLUSTER_0_MACHINE_TYPE="e2-standard-4"
15export K8S_CLUSTER_0_CONTEXT_NAME="gke_${MDB_GKE_PROJECT}_${K8S_CLUSTER_0_ZONE}_${K8S_CLUSTER_0}"
16
17export K8S_CLUSTER_1="k8s-mdb-1"
18export K8S_CLUSTER_1_ZONE="europe-central2-b"
19export K8S_CLUSTER_1_NUMBER_OF_NODES=3
20export K8S_CLUSTER_1_MACHINE_TYPE="e2-standard-4"
21export K8S_CLUSTER_1_CONTEXT_NAME="gke_${MDB_GKE_PROJECT}_${K8S_CLUSTER_1_ZONE}_${K8S_CLUSTER_1}"
22
23export K8S_CLUSTER_2="k8s-mdb-2"
24export K8S_CLUSTER_2_ZONE="europe-central2-c"
25export K8S_CLUSTER_2_NUMBER_OF_NODES=1
26export K8S_CLUSTER_2_MACHINE_TYPE="e2-standard-4"
27export K8S_CLUSTER_2_CONTEXT_NAME="gke_${MDB_GKE_PROJECT}_${K8S_CLUSTER_2_ZONE}_${K8S_CLUSTER_2}"
28
29# Comment out the following line so that the script does not create preemptible nodes.
30# DO NOT USE preemptible nodes in production.
31export GKE_SPOT_INSTANCES_SWITCH="--preemptible"
32
33export S3_OPLOG_BUCKET_NAME=s3-oplog-store
34export S3_SNAPSHOT_BUCKET_NAME=s3-snapshot-store
35
36# minio defaults
37export S3_ENDPOINT="minio.tenant-tiny.svc.cluster.local"
38export S3_ACCESS_KEY="console"
39export S3_SECRET_KEY="console123"
40
41export OPERATOR_HELM_CHART="mongodb/enterprise-operator"
42
43# (Optional) Change the following setting when using the external URL.
44# This env variable is used in OpenSSL configuration to generate
45# server certificates for Ops Manager Application.
46export OPS_MANAGER_EXTERNAL_DOMAIN="om-svc.${NAMESPACE}.svc.cluster.local"
47
48export OPS_MANAGER_VERSION="7.0.4"
49export APPDB_VERSION="7.0.9-ubi8"

Adjust the settings in the previous example for your needs as instructed in the comments and source them into your shell as follows:

source env_variables.sh

Important

Each time after you update env_variables.sh, run source env_variables.sh to ensure that the scripts in this section use updated variables.

This procedure applies to deploying an Ops Manager instance on multiple Kubernetes clusters.

1

You may skip this step if you already have installed and configured your own Kubernetes clusters with a service mesh.

  1. Create three GKE (Google Kubernetes Engine) clusters:

    1gcloud container clusters create "${K8S_CLUSTER_0}" \
    2 --zone="${K8S_CLUSTER_0_ZONE}" \
    3 --num-nodes="${K8S_CLUSTER_0_NUMBER_OF_NODES}" \
    4 --machine-type "${K8S_CLUSTER_0_MACHINE_TYPE}" \
    5 ${GKE_SPOT_INSTANCES_SWITCH:-""}
    1gcloud container clusters create "${K8S_CLUSTER_1}" \
    2 --zone="${K8S_CLUSTER_1_ZONE}" \
    3 --num-nodes="${K8S_CLUSTER_1_NUMBER_OF_NODES}" \
    4 --machine-type "${K8S_CLUSTER_1_MACHINE_TYPE}" \
    5 ${GKE_SPOT_INSTANCES_SWITCH:-""}
    1gcloud container clusters create "${K8S_CLUSTER_2}" \
    2 --zone="${K8S_CLUSTER_2_ZONE}" \
    3 --num-nodes="${K8S_CLUSTER_2_NUMBER_OF_NODES}" \
    4 --machine-type "${K8S_CLUSTER_2_MACHINE_TYPE}" \
    5 ${GKE_SPOT_INSTANCES_SWITCH:-""}
  2. Obtain credentials and save contexts to the current kubeconfig file. By default, this file is located in the ~/.kube/config directory and referenced by the $KUBECONFIG environment variable.

    1gcloud container clusters get-credentials "${K8S_CLUSTER_0}" --zone="${K8S_CLUSTER_0_ZONE}"
    2gcloud container clusters get-credentials "${K8S_CLUSTER_1}" --zone="${K8S_CLUSTER_1_ZONE}"
    3gcloud container clusters get-credentials "${K8S_CLUSTER_2}" --zone="${K8S_CLUSTER_2_ZONE}"

    All kubectl commands reference these contexts using the following variables:

    • $K8S_CLUSTER_0_CONTEXT_NAME

    • $K8S_CLUSTER_1_CONTEXT_NAME

    • $K8S_CLUSTER_2_CONTEXT_NAME

  3. Verify that kubectl has access to Kubernetes clusters:

    1echo "Nodes in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
    2kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" get nodes
    3echo; echo "Nodes in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
    4kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" get nodes
    5echo; echo "Nodes in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"
    6kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" get nodes
  4. Install Istio service mesh to allow cross-cluster DNS resolution and network connectivity between Kubernetes clusters:

    1CTX_CLUSTER1=${K8S_CLUSTER_0_CONTEXT_NAME} \
    2CTX_CLUSTER2=${K8S_CLUSTER_1_CONTEXT_NAME} \
    3CTX_CLUSTER3=${K8S_CLUSTER_2_CONTEXT_NAME} \
    4ISTIO_VERSION="1.20.2" \
    5../multi-cluster/install_istio_separate_network.sh
2

Note

To enable sidecar injection in Istio, the following commands add the istio-injection=enabled labels to the $OPERATOR_NAMESPACE and the mongodb namespaces on each member cluster. If you use another service mesh, configure it to handle network traffic in the created namespaces.

  • Create a separate namespace, mongodb-operator, referenced by the $OPERATOR_NAMESPACE environment variable for the Kubernetes Operator deployment.

  • Create the same $OPERATOR_NAMESPACE on each member Kubernetes cluster. This is needed so that the kubectl mongodb plugin can create a service account for the Kubernetes Operator on each member cluster. The Kubernetes Operator uses these service accounts on the operator cluster to perform operations on each member cluster.

    1kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
    2kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite
    3
    4kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
    5kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite
    6
    7kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
    8kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite
  • On each member cluster, including the member cluster that serves as the operator cluster, create another, separate namespace, mongodb. The Kubernetes Operator uses this namespace for Ops Manager resources and components.

    1kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" create namespace "${NAMESPACE}"
    2kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite
    3
    4kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" create namespace "${NAMESPACE}"
    5kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite
    6
    7kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" create namespace "${NAMESPACE}"
    8kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite
3

This step is optional if you use official Helm charts and images from the Quay registry.

1kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" create secret generic "image-registries-secret" \
2 --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
3kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
4 --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
5kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
6 --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
7kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
8 --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
4

The following optional scripts verify whether the service mesh is configured correctly for cross-cluster DNS resolution and connectivity.

  1. Run this script on cluster 0:

    1kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2 apiVersion: apps/v1
    3 kind: StatefulSet
    4 metadata:
    5 name: echoserver0
    6 spec:
    7 replicas: 1
    8 selector:
    9 matchLabels:
    10 app: echoserver0
    11 template:
    12 metadata:
    13 labels:
    14 app: echoserver0
    15 spec:
    16 containers:
    17 - image: k8s.gcr.io/echoserver:1.10
    18 imagePullPolicy: Always
    19 name: echoserver0
    20 ports:
    21 - containerPort: 8080
    22EOF
  2. Run this script on cluster 1:

    1kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2 apiVersion: apps/v1
    3 kind: StatefulSet
    4 metadata:
    5 name: echoserver1
    6 spec:
    7 replicas: 1
    8 selector:
    9 matchLabels:
    10 app: echoserver1
    11 template:
    12 metadata:
    13 labels:
    14 app: echoserver1
    15 spec:
    16 containers:
    17 - image: k8s.gcr.io/echoserver:1.10
    18 imagePullPolicy: Always
    19 name: echoserver1
    20 ports:
    21 - containerPort: 8080
    22EOF
  3. Run this script on cluster 2:

    1kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2 apiVersion: apps/v1
    3 kind: StatefulSet
    4 metadata:
    5 name: echoserver2
    6 spec:
    7 replicas: 1
    8 selector:
    9 matchLabels:
    10 app: echoserver2
    11 template:
    12 metadata:
    13 labels:
    14 app: echoserver2
    15 spec:
    16 containers:
    17 - image: k8s.gcr.io/echoserver:1.10
    18 imagePullPolicy: Always
    19 name: echoserver2
    20 ports:
    21 - containerPort: 8080
    22EOF
  4. Run this script to wait for the creation of StatefulSets:

    1kubectl wait --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver0-0 --timeout=60s
    2kubectl wait --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver1-0 --timeout=60s
    3kubectl wait --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver2-0 --timeout=60s
  5. Create Pod service on cluster 0:

    1kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2apiVersion: v1
    3kind: Service
    4metadata:
    5 name: echoserver0-0
    6spec:
    7 ports:
    8 - port: 8080
    9 targetPort: 8080
    10 protocol: TCP
    11 selector:
    12 statefulset.kubernetes.io/pod-name: "echoserver0-0"
    13EOF
  6. Create Pod service on cluster 1:

    1kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2apiVersion: v1
    3kind: Service
    4metadata:
    5 name: echoserver1-0
    6spec:
    7 ports:
    8 - port: 8080
    9 targetPort: 8080
    10 protocol: TCP
    11 selector:
    12 statefulset.kubernetes.io/pod-name: "echoserver1-0"
    13EOF
  7. Create Pod service on cluster 2:

    1kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2apiVersion: v1
    3kind: Service
    4metadata:
    5 name: echoserver2-0
    6spec:
    7 ports:
    8 - port: 8080
    9 targetPort: 8080
    10 protocol: TCP
    11 selector:
    12 statefulset.kubernetes.io/pod-name: "echoserver2-0"
    13EOF
  8. Create round robin service on cluster 0:

    1kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2apiVersion: v1
    3kind: Service
    4metadata:
    5 name: echoserver
    6spec:
    7 ports:
    8 - port: 8080
    9 targetPort: 8080
    10 protocol: TCP
    11 selector:
    12 app: echoserver0
    13EOF
  9. Create round robin service on cluster 1:

    1kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2apiVersion: v1
    3kind: Service
    4metadata:
    5 name: echoserver
    6spec:
    7 ports:
    8 - port: 8080
    9 targetPort: 8080
    10 protocol: TCP
    11 selector:
    12 app: echoserver1
    13EOF
  10. Create round robin service on cluster 2:

    1kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2apiVersion: v1
    3kind: Service
    4metadata:
    5 name: echoserver
    6spec:
    7 ports:
    8 - port: 8080
    9 targetPort: 8080
    10 protocol: TCP
    11 selector:
    12 app: echoserver2
    13EOF
  11. Verify Pod 0 from cluster 1:

    1source_cluster=${K8S_CLUSTER_1_CONTEXT_NAME}
    2target_pod="echoserver0-0"
    3source_pod="echoserver1-0"
    4target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7 /bin/bash -c "curl -v ${target_url}" 2>&1);
    8grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)
  12. Verify Pod 1 from cluster 0:

    1source_cluster=${K8S_CLUSTER_0_CONTEXT_NAME}
    2target_pod="echoserver1-0"
    3source_pod="echoserver0-0"
    4target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7 /bin/bash -c "curl -v ${target_url}" 2>&1);
    8grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)
  13. Verify Pod 1 from cluster 2:

    1source_cluster=${K8S_CLUSTER_2_CONTEXT_NAME}
    2target_pod="echoserver1-0"
    3source_pod="echoserver2-0"
    4target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7 /bin/bash -c "curl -v ${target_url}" 2>&1);
    8grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)
  1. Verify Pod 2 from cluster 0:

    1source_cluster=${K8S_CLUSTER_0_CONTEXT_NAME}
    2target_pod="echoserver2-0"
    3source_pod="echoserver0-0"
    4target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7 /bin/bash -c "curl -v ${target_url}" 2>&1);
    8grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)
  2. Run the cleanup script:

    1kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver0
    2kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver1
    3kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver2
    4kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    5kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    6kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    7kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver0-0
    8kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver1-0
    9kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver2-0
5

In this step, you use the kubectl mongodb plugin to automate the Kubernetes cluster configuration that is necessary for the Kubernetes Operator to manage workloads on multiple Kubernetes clusters.

Because you configure the Kubernetes clusters before you install the Kubernetes Operator, when you deploy the Kubernetes Operator for the multi-Kubernetes cluster operation, all the necessary multi-cluster configuration is already in place.

As stated in the Overview, the Kubernetes Operator has the configuration for three member clusters that you can use to deploy Ops Manager MongoDB databases. The first cluster is also used as the operator cluster, where you install the Kubernetes Operator and deploy the custom resources.

1kubectl mongodb multicluster setup \
2 --central-cluster="${K8S_CLUSTER_0_CONTEXT_NAME}" \
3 --member-clusters="${K8S_CLUSTER_0_CONTEXT_NAME},${K8S_CLUSTER_1_CONTEXT_NAME},${K8S_CLUSTER_2_CONTEXT_NAME}" \
4 --member-cluster-namespace="${NAMESPACE}" \
5 --central-cluster-namespace="${OPERATOR_NAMESPACE}" \
6 --create-service-account-secrets \
7 --install-database-roles=true \
8 --image-pull-secrets=image-registries-secret
6
  1. Install the Kubernetes Operator into the $OPERATOR_NAMESPACE, configured to watch $NAMESPACE and to manage three member Kubernetes clusters. At this point in the procedure, ServiceAccounts and roles are already deployed by the kubectl mongodb plugin. Therefore, the following scripts skip configuring them and set operator.createOperatorServiceAccount=false and operator.createResourcesServiceAccountsAndRoles=false. The scripts specify the multiCluster.clusters setting to instruct the Helm chart to deploy the Kubernetes Operator in multi-cluster mode.

    1helm upgrade --install \
    2 --debug \
    3 --kube-context "${K8S_CLUSTER_0_CONTEXT_NAME}" \
    4 mongodb-enterprise-operator-multi-cluster \
    5 "${OPERATOR_HELM_CHART}" \
    6 --namespace="${OPERATOR_NAMESPACE}" \
    7 --set namespace="${OPERATOR_NAMESPACE}" \
    8 --set operator.namespace="${OPERATOR_NAMESPACE}" \
    9 --set operator.watchNamespace="${NAMESPACE}" \
    10 --set operator.name=mongodb-enterprise-operator-multi-cluster \
    11 --set operator.createOperatorServiceAccount=false \
    12 --set operator.createResourcesServiceAccountsAndRoles=false \
    13 --set "multiCluster.clusters={${K8S_CLUSTER_0_CONTEXT_NAME},${K8S_CLUSTER_1_CONTEXT_NAME},${K8S_CLUSTER_2_CONTEXT_NAME}}" \
    14 --set "${OPERATOR_ADDITIONAL_HELM_VALUES:-"dummy=value"}"
  2. Check the Kubernetes Operator deployment:

    1kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" rollout status deployment/mongodb-enterprise-operator-multi-cluster
    2echo "Operator deployment in ${OPERATOR_NAMESPACE} namespace"
    3kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" get deployments
    4echo; echo "Operator pod in ${OPERATOR_NAMESPACE} namespace"
    5kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" get pods
7

In this step, you enable TLS for the Application Database and the Ops Manager Application. If you don't want to use TLS, remove the following fields from the MongoDBOpsManager resources:

  1. Optional. Generate keys and certificates:

    Use the openssl command line tool to generate self-signed CAs and certificates for testing purposes.

    1mkdir certs || true
    2
    3cat <<EOF >certs/ca.cnf
    4[ req ]
    5default_bits = 2048
    6prompt = no
    7default_md = sha256
    8distinguished_name = dn
    9
    10[ dn ]
    11C=US
    12ST=New York
    13L=New York
    14O=Example Company
    15OU=IT Department
    16CN=exampleCA
    17EOF
    18
    19cat <<EOF >certs/om.cnf
    20[ req ]
    21default_bits = 2048
    22prompt = no
    23default_md = sha256
    24distinguished_name = dn
    25req_extensions = req_ext
    26
    27[ dn ]
    28C=US
    29ST=New York
    30L=New York
    31O=Example Company
    32OU=IT Department
    33CN=${OPS_MANAGER_EXTERNAL_DOMAIN}
    34
    35[ req_ext ]
    36subjectAltName = @alt_names
    37keyUsage = critical, digitalSignature, keyEncipherment
    38extendedKeyUsage = serverAuth, clientAuth
    39
    40[ alt_names ]
    41DNS.1 = ${OPS_MANAGER_EXTERNAL_DOMAIN}
    42DNS.2 = om-svc.${NAMESPACE}.svc.cluster.local
    43EOF
    44
    45cat <<EOF >certs/appdb.cnf
    46[ req ]
    47default_bits = 2048
    48prompt = no
    49default_md = sha256
    50distinguished_name = dn
    51req_extensions = req_ext
    52
    53[ dn ]
    54C=US
    55ST=New York
    56L=New York
    57O=Example Company
    58OU=IT Department
    59CN=AppDB
    60
    61[ req_ext ]
    62subjectAltName = @alt_names
    63keyUsage = critical, digitalSignature, keyEncipherment
    64extendedKeyUsage = serverAuth, clientAuth
    65
    66[ alt_names ]
    67# multi-cluster mongod hostnames from service for each pod
    68DNS.1 = *.${NAMESPACE}.svc.cluster.local
    69# single-cluster mongod hostnames from headless service
    70DNS.2 = *.om-db-svc.${NAMESPACE}.svc.cluster.local
    71EOF
    72
    73# generate CA keypair and certificate
    74openssl genrsa -out certs/ca.key 2048
    75openssl req -x509 -new -nodes -key certs/ca.key -days 1024 -out certs/ca.crt -config certs/ca.cnf
    76
    77# generate OpsManager's keypair and certificate
    78openssl genrsa -out certs/om.key 2048
    79openssl req -new -key certs/om.key -out certs/om.csr -config certs/om.cnf
    80
    81# generate AppDB's keypair and certificate
    82openssl genrsa -out certs/appdb.key 2048
    83openssl req -new -key certs/appdb.key -out certs/appdb.csr -config certs/appdb.cnf
    84
    85# generate certificates signed by CA for OpsManager and AppDB
    86openssl x509 -req -in certs/om.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/om.crt -days 365 -sha256 -extfile certs/om.cnf -extensions req_ext
    87openssl x509 -req -in certs/appdb.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/appdb.crt -days 365 -sha256 -extfile certs/appdb.cnf -extensions req_ext
  2. Create secrets with TLS keys:

    If you prefer to use your own keys and certificates, skip the previous generation step and put the keys and certificates into the following files:

    • certs/ca.crt - CA certificates. These are not necessary when using trusted certificates.

    • certs/appdb.key - private key for the Application Database.

    • certs/appdb.crt - certificate for the Application Database.

    • certs/om.key - private key for Ops Manager.

    • certs/om.crt - certificate for Ops Manager.

    1kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret tls cert-prefix-om-cert \
    2 --cert=certs/om.crt \
    3 --key=certs/om.key
    4
    5kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret tls cert-prefix-om-db-cert \
    6 --cert=certs/appdb.crt \
    7 --key=certs/appdb.key
    8
    9kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create configmap om-cert-ca --from-file="mms-ca.crt=certs/ca.crt"
    10kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create configmap appdb-cert-ca --from-file="ca-pem=certs/ca.crt"
8

At this point, you have prepared the environment and the Kubernetes Operator to deploy the Ops Manager resource.

  1. Create the necessary credentials for the Ops Manager admin user that the Kubernetes Operator will create after deploying the Ops Manager Application instance:

    1kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" --namespace "${NAMESPACE}" create secret generic om-admin-user-credentials \
    2 --from-literal=Username="admin" \
    3 --from-literal=Password="Passw0rd@" \
    4 --from-literal=FirstName="Jane" \
    5 --from-literal=LastName="Doe"
  2. Deploy the simplest MongoDBOpsManager custom resource possible (with TLS enabled) on a single member cluster, which is also known as the operator cluster.

    This deployment is almost the same as for the single-cluster mode, but with spec.topology and spec.applicationDatabase.topology set to MultiCluster.

    Deploying this way shows that a single Kubernetes cluster deployment is a special case of a multi-Kubernetes cluster deployment on a single Kubernetes member cluster. You can start deploying the Ops Manager Application and the Application Database on as many Kubernetes clusters as necessary from the beginning, and don't have to start with the deployment with only a single member Kubernetes cluster.

    At this point, you have prepared the Ops Manager deployment to span more than one Kubernetes cluster, which you will do later in this procedure.

    1kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2apiVersion: mongodb.com/v1
    3kind: MongoDBOpsManager
    4metadata:
    5 name: om
    6spec:
    7 topology: MultiCluster
    8 version: "${OPS_MANAGER_VERSION}"
    9 adminCredentials: om-admin-user-credentials
    10 security:
    11 certsSecretPrefix: cert-prefix
    12 tls:
    13 ca: om-cert-ca
    14 clusterSpecList:
    15 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
    16 members: 1
    17 applicationDatabase:
    18 version: "${APPDB_VERSION}"
    19 topology: MultiCluster
    20 security:
    21 certsSecretPrefix: cert-prefix
    22 tls:
    23 ca: appdb-cert-ca
    24 clusterSpecList:
    25 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
    26 members: 3
    27 backup:
    28 enabled: false
    29EOF
  3. Wait for the Kubernetes Operator to pick up the work and reach the status.applicationDatabase.phase=Pending state. Wait for both the Application Database and Ops Manager deployments to complete.

    1echo "Waiting for Application Database to reach Pending phase..."
    2kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s
  4. Deploy Ops Manager. The Kubernetes Operator deploys Ops Manager by performing the following steps. It:

    • Deploys the Application Database's replica set nodes and waits for the MongoDB processes in the replica set to start running.

    • Deploys the Ops Manager Application instance with the Application Database's connection string and waits for it to become ready.

    • Adds the Monitoring MongoDB Agent containers to each Application Database's Pod.

    • Waits for both the Ops Manager Application and the Application Database Pods to start running.

    1echo "Waiting for Application Database to reach Running phase..."
    2kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
    3echo; echo "Waiting for Ops Manager to reach Running phase..."
    4kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s
    5echo; echo "Waiting for Application Database to reach Pending phase (enabling monitoring)..."
    6kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
    7echo "Waiting for Application Database to reach Running phase..."
    8kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
    9echo; echo "Waiting for Ops Manager to reach Running phase..."
    10kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s
    11echo; echo "MongoDBOpsManager resource"
    12kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
    13echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
    14kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
    15echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
    16kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

    Now that you have deployed a single-member cluster in a multi-cluster mode, you can reconfigure this deployment to span more than one Kubernetes cluster.

  5. On the second member cluster, deploy two additional Application Database replica set members and one additional instance of the Ops Manager Application:

    1kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2apiVersion: mongodb.com/v1
    3kind: MongoDBOpsManager
    4metadata:
    5 name: om
    6spec:
    7 topology: MultiCluster
    8 version: "${OPS_MANAGER_VERSION}"
    9 adminCredentials: om-admin-user-credentials
    10 security:
    11 certsSecretPrefix: cert-prefix
    12 tls:
    13 ca: om-cert-ca
    14 clusterSpecList:
    15 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
    16 members: 1
    17 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
    18 members: 1
    19 applicationDatabase:
    20 version: "${APPDB_VERSION}"
    21 topology: MultiCluster
    22 security:
    23 certsSecretPrefix: cert-prefix
    24 tls:
    25 ca: appdb-cert-ca
    26 clusterSpecList:
    27 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
    28 members: 3
    29 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
    30 members: 2
    31 backup:
    32 enabled: false
    33EOF
  6. Wait for the Kubernetes Operator to pick up the work (pending phase):

    1echo "Waiting for Application Database to reach Pending phase..."
    2kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s
  7. Wait for the Kubernetes Operator to finish deploying all components:

    1echo "Waiting for Application Database to reach Running phase..."
    2kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s
    3echo; echo "Waiting for Ops Manager to reach Running phase..."
    4kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s
    5echo; echo "MongoDBOpsManager resource"
    6kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
    7echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
    8kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
    9echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
    10kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
9

In a multi-Kubernetes cluster deployment of the Ops Manager Application, you can configure only S3-based backup storage. This procedure refers to S3_* defined in env_variables.sh.

  1. Optional. Install the MinIO Operator.

    This procedure deploys S3-compatible storage for your backups using the MinIO Operator. You can skip this step if you have AWS S3 or other S3-compatible buckets available. Adjust the S3_* variables accordingly in env_variables.sh in this case.

    1kustomize build "github.com/minio/operator/resources/?timeout=120&ref=v5.0.12" | \
    2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
    3
    4kustomize build "github.com/minio/operator/examples/kustomization/tenant-tiny?timeout=120&ref=v5.0.12" | \
    5 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
    6
    7# add two buckets to the tenant config
    8kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "tenant-tiny" patch tenant/myminio \
    9 --type='json' \
    10 -p="[{\"op\": \"add\", \"path\": \"/spec/buckets\", \"value\": [{\"name\": \"${S3_OPLOG_BUCKET_NAME}\"}, {\"name\": \"${S3_SNAPSHOT_BUCKET_NAME}\"}]}]"
  2. Before you configure and enable backup, create secrets:

    • s3-access-secret - contains S3 credentials.

    • s3-ca-cert - contains a CA certificate that issued the bucket's server certificate. In the case of the sample MinIO deployment used in this procedure, the default Kubernetes Root CA certificate is used to sign the certificate. Because it's not a publicly trusted CA certificate, you must provide it so that Ops Manager can trust the connection.

    If you use publicly trusted certificates, you may skip this step and remove the values from the spec.backup.s3Stores.customCertificateSecretRefs and spec.backup.s3OpLogStores.customCertificateSecretRefs settings.

    1kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic s3-access-secret \
    2 --from-literal=accessKey="${S3_ACCESS_KEY}" \
    3 --from-literal=secretKey="${S3_SECRET_KEY}"
    4
    5# minio TLS secrets are signed with the default k8s root CA
    6kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic s3-ca-cert \
    7 --from-literal=ca.crt="$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n kube-system get configmap kube-root-ca.crt -o jsonpath="{.data.ca\.crt}")"
10
  1. The Kubernetes Operator can configure and deploy all components, the Ops Manager Application, the Backup Daemon instances, and the Application Database's replica set nodes in any combination on any member clusters for which you configure the Kubernetes Operator.

    To illustrate the flexibility of the multi-Kubernetes cluster deployment configuration, deploy only one Backup Daemon instance on the third member cluster and specify zero Backup Daemon members for the first and second clusters.

    1kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2apiVersion: mongodb.com/v1
    3kind: MongoDBOpsManager
    4metadata:
    5 name: om
    6spec:
    7 topology: MultiCluster
    8 version: "${OPS_MANAGER_VERSION}"
    9 adminCredentials: om-admin-user-credentials
    10 security:
    11 certsSecretPrefix: cert-prefix
    12 tls:
    13 ca: om-cert-ca
    14 clusterSpecList:
    15 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
    16 members: 1
    17 backup:
    18 members: 0
    19 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
    20 members: 1
    21 backup:
    22 members: 0
    23 - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}"
    24 members: 0
    25 backup:
    26 members: 1
    27 configuration: # to avoid configuration wizard on first login
    28 mms.adminEmailAddr: email@example.com
    29 mms.fromEmailAddr: email@example.com
    30 mms.ignoreInitialUiSetup: "true"
    31 mms.mail.hostname: smtp@example.com
    32 mms.mail.port: "465"
    33 mms.mail.ssl: "true"
    34 mms.mail.transport: smtp
    35 mms.minimumTLSVersion: TLSv1.2
    36 mms.replyToEmailAddr: email@example.com
    37 applicationDatabase:
    38 version: "${APPDB_VERSION}"
    39 topology: MultiCluster
    40 security:
    41 certsSecretPrefix: cert-prefix
    42 tls:
    43 ca: appdb-cert-ca
    44 clusterSpecList:
    45 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
    46 members: 3
    47 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
    48 members: 2
    49 backup:
    50 enabled: true
    51 s3Stores:
    52 - name: my-s3-block-store
    53 s3SecretRef:
    54 name: "s3-access-secret"
    55 pathStyleAccessEnabled: true
    56 s3BucketEndpoint: "${S3_ENDPOINT}"
    57 s3BucketName: "${S3_SNAPSHOT_BUCKET_NAME}"
    58 customCertificateSecretRefs:
    59 - name: s3-ca-cert
    60 key: ca.crt
    61 s3OpLogStores:
    62 - name: my-s3-oplog-store
    63 s3SecretRef:
    64 name: "s3-access-secret"
    65 s3BucketEndpoint: "${S3_ENDPOINT}"
    66 s3BucketName: "${S3_OPLOG_BUCKET_NAME}"
    67 pathStyleAccessEnabled: true
    68 customCertificateSecretRefs:
    69 - name: s3-ca-cert
    70 key: ca.crt
    71EOF
  2. Wait until the Kubernetes Operator finishes its configuration:

    1echo; echo "Waiting for Backup to reach Running phase..."
    2kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.backup.phase}'=Running opsmanager/om --timeout=1200s
    3echo "Waiting for Application Database to reach Running phase..."
    4kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s
    5echo; echo "Waiting for Ops Manager to reach Running phase..."
    6kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s
    7echo; echo "MongoDBOpsManager resource"
    8kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
    9echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
    10kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
    11echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
    12kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
    13echo; echo "Pods running in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"
    14kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
11

Run the following script to delete the GKE clusters and clean up your environment.

Important

The following commands are not reversible. They delete all clusters referenced in env_variables.sh. Don't run these commands if you wish to retain the GKE clusters, for example, if you didn't create the GKE clusters as part of this procedure.

1yes | gcloud container clusters delete "${K8S_CLUSTER_0}" --zone="${K8S_CLUSTER_0_ZONE}" &
2yes | gcloud container clusters delete "${K8S_CLUSTER_1}" --zone="${K8S_CLUSTER_1_ZONE}" &
3yes | gcloud container clusters delete "${K8S_CLUSTER_2}" --zone="${K8S_CLUSTER_2_ZONE}" &
4wait

Back

Plan Your Ops Manager Resource

Next

Deploy an