Deploy Ops Manager Resources on Multiple Kubernetes Clusters

Create Kubernetes clusters.

You may skip this step if you already have installed and configured your own Kubernetes clusters with a service mesh.

1. Create three GKE (Google Kubernetes Engine) clusters:

   1
   gcloud container clusters create "${K8S_CLUSTER_0}" \
   2
         --zone="${K8S_CLUSTER_0_ZONE}" \
   3
         --num-nodes="${K8S_CLUSTER_0_NUMBER_OF_NODES}" \
   4
         --machine-type "${K8S_CLUSTER_0_MACHINE_TYPE}" \
   5
         ${GKE_SPOT_INSTANCES_SWITCH:-""}

   1
   gcloud container clusters create "${K8S_CLUSTER_1}" \
   2
         --zone="${K8S_CLUSTER_1_ZONE}" \
   3
         --num-nodes="${K8S_CLUSTER_1_NUMBER_OF_NODES}" \
   4
         --machine-type "${K8S_CLUSTER_1_MACHINE_TYPE}" \
   5
         ${GKE_SPOT_INSTANCES_SWITCH:-""}

   1
   gcloud container clusters create "${K8S_CLUSTER_2}" \
   2
         --zone="${K8S_CLUSTER_2_ZONE}" \
   3
         --num-nodes="${K8S_CLUSTER_2_NUMBER_OF_NODES}" \
   4
         --machine-type "${K8S_CLUSTER_2_MACHINE_TYPE}" \
   5
         ${GKE_SPOT_INSTANCES_SWITCH:-""}

2. Set your default gcloud project:

   1
   gcloud config set project "${MDB_GKE_PROJECT}"

3. Obtain credentials and save contexts to the current kubeconfig file. By default, this file is located in the ~/.kube/config directory and referenced by the $KUBECONFIG environment variable.

   1
   gcloud container clusters get-credentials "${K8S_CLUSTER_0}" --zone="${K8S_CLUSTER_0_ZONE}"
   2
   gcloud container clusters get-credentials "${K8S_CLUSTER_1}" --zone="${K8S_CLUSTER_1_ZONE}"
   3
   gcloud container clusters get-credentials "${K8S_CLUSTER_2}" --zone="${K8S_CLUSTER_2_ZONE}"

   All kubectl commands reference these contexts using the following variables:

   • $K8S_CLUSTER_0_CONTEXT_NAME

   • $K8S_CLUSTER_1_CONTEXT_NAME

   • $K8S_CLUSTER_2_CONTEXT_NAME

4. Verify that kubectl has access to Kubernetes clusters:

   1
   echo "Nodes in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" get nodes
   3
   echo; echo "Nodes in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   4
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" get nodes
   5
   echo; echo "Nodes in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"
   6
   kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" get nodes

   VIEW OUTPUT

5. Install Istio service mesh to allow cross-cluster DNS resolution and network connectivity between Kubernetes clusters:

   1
   CTX_CLUSTER1=${K8S_CLUSTER_0_CONTEXT_NAME} \
   2
   CTX_CLUSTER2=${K8S_CLUSTER_1_CONTEXT_NAME} \
   3
   CTX_CLUSTER3=${K8S_CLUSTER_2_CONTEXT_NAME} \
   4
   ISTIO_VERSION="1.20.2" \
   5
   ../multi-cluster/install_istio_separate_network.sh

Create namespaces.

• Create a separate namespace, mongodb-operator, referenced by the $OPERATOR_NAMESPACE environment variable for the Kubernetes Operator deployment.

• Create the same $OPERATOR_NAMESPACE on each member Kubernetes cluster. This is needed so that the kubectl mongodb plugin can create a service account for the Kubernetes Operator on each member cluster. The Kubernetes Operator uses these service accounts on the operator cluster to perform operations on each member cluster.

  1
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
  2
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite
  3
  4
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
  5
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite
  6
  7
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
  8
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite

• On each member cluster, including the member cluster that serves as the operator cluster, create another, separate namespace, mongodb. The Kubernetes Operator uses this namespace for Ops Manager resources and components.

  1
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" create namespace "${NAMESPACE}"
  2
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite
  3
  4
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" create namespace "${NAMESPACE}"
  5
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite
  6
  7
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" create namespace "${NAMESPACE}"
  8
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite

Optional. Authorize clusters to pull secrets from private image registries.

This step is optional if you use official Helm charts and images from the Quay registry.

1
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" create secret generic "image-registries-secret" \
2
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
3
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
4
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
5
kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
6
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
7
kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
8
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson

Optional. Check cluster connectivity.

The following optional scripts verify whether the service mesh is configured correctly for cross-cluster DNS resolution and connectivity.

1. Run this script on cluster 0:

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
     apiVersion: apps/v1
   3
     kind: StatefulSet
   4
     metadata:
   5
       name: echoserver0
   6
     spec:
   7
       replicas: 1
   8
       selector:
   9
         matchLabels:
   10
           app: echoserver0
   11
       template:
   12
         metadata:
   13
           labels:
   14
             app: echoserver0
   15
         spec:
   16
           containers:
   17
             - image: k8s.gcr.io/echoserver:1.10
   18
               imagePullPolicy: Always
   19
               name: echoserver0
   20
               ports:
   21
                 - containerPort: 8080
   22
   EOF

2. Run this script on cluster 1:

   1
   kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
     apiVersion: apps/v1
   3
     kind: StatefulSet
   4
     metadata:
   5
       name: echoserver1
   6
     spec:
   7
       replicas: 1
   8
       selector:
   9
         matchLabels:
   10
           app: echoserver1
   11
       template:
   12
         metadata:
   13
           labels:
   14
             app: echoserver1
   15
         spec:
   16
           containers:
   17
             - image: k8s.gcr.io/echoserver:1.10
   18
               imagePullPolicy: Always
   19
               name: echoserver1
   20
               ports:
   21
                 - containerPort: 8080
   22
   EOF

3. Run this script on cluster 2:

   1
   kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
     apiVersion: apps/v1
   3
     kind: StatefulSet
   4
     metadata:
   5
       name: echoserver2
   6
     spec:
   7
       replicas: 1
   8
       selector:
   9
         matchLabels:
   10
           app: echoserver2
   11
       template:
   12
         metadata:
   13
           labels:
   14
             app: echoserver2
   15
         spec:
   16
           containers:
   17
             - image: k8s.gcr.io/echoserver:1.10
   18
               imagePullPolicy: Always
   19
               name: echoserver2
   20
               ports:
   21
                 - containerPort: 8080
   22
   EOF

4. Run this script to wait for the creation of StatefulSets:

   1
   kubectl wait --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver0-0 --timeout=60s
   2
   kubectl wait --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver1-0 --timeout=60s
   3
   kubectl wait --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver2-0 --timeout=60s

5. Create Pod service on cluster 0:

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver0-0
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       statefulset.kubernetes.io/pod-name: "echoserver0-0"
   13
   EOF

6. Create Pod service on cluster 1:

   1
   kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver1-0
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       statefulset.kubernetes.io/pod-name: "echoserver1-0"
   13
   EOF

7. Create Pod service on cluster 2:

   1
   kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver2-0
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       statefulset.kubernetes.io/pod-name: "echoserver2-0"
   13
   EOF

8. Create round robin service on cluster 0:

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       app: echoserver0
   13
   EOF

9. Create round robin service on cluster 1:

   1
   kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       app: echoserver1
   13
   EOF

10. Create round robin service on cluster 2:

    1
    kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2
    apiVersion: v1
    3
    kind: Service
    4
    metadata:
    5
      name: echoserver
    6
    spec:
    7
      ports:
    8
        - port: 8080
    9
          targetPort: 8080
    10
          protocol: TCP
    11
      selector:
    12
        app: echoserver2
    13
    EOF

11. Verify Pod 0 from cluster 1:

    1
    source_cluster=${K8S_CLUSTER_1_CONTEXT_NAME}
    2
    target_pod="echoserver0-0"
    3
    source_pod="echoserver1-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    VIEW OUTPUT

12. Verify Pod 1 from cluster 0:

    1
    source_cluster=${K8S_CLUSTER_0_CONTEXT_NAME}
    2
    target_pod="echoserver1-0"
    3
    source_pod="echoserver0-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    VIEW OUTPUT

13. Verify Pod 1 from cluster 2:

    1
    source_cluster=${K8S_CLUSTER_2_CONTEXT_NAME}
    2
    target_pod="echoserver1-0"
    3
    source_pod="echoserver2-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    VIEW OUTPUT

15. Verify Pod 2 from cluster 0:

    1
    source_cluster=${K8S_CLUSTER_0_CONTEXT_NAME}
    2
    target_pod="echoserver2-0"
    3
    source_pod="echoserver0-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    VIEW OUTPUT

16. Run the cleanup script:

    1
    kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver0
    2
    kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver1
    3
    kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver2
    4
    kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    5
    kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    6
    kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    7
    kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver0-0
    8
    kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver1-0
    9
    kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver2-0

Deploy a multi-cluster configuration.

In this step, you use the kubectl mongodb plugin to automate the Kubernetes cluster configuration that is necessary for the Kubernetes Operator to manage workloads on multiple Kubernetes clusters.

Because you configure the Kubernetes clusters before you install the Kubernetes Operator, when you deploy the Kubernetes Operator for the multi-Kubernetes cluster operation, all the necessary multi-cluster configuration is already in place.

As stated in the Overview, the Kubernetes Operator has the configuration for three member clusters that you can use to deploy Ops Manager MongoDB databases. The first cluster is also used as the operator cluster, where you install the Kubernetes Operator and deploy the custom resources.

1
kubectl mongodb multicluster setup \
2
  --central-cluster="${K8S_CLUSTER_0_CONTEXT_NAME}" \
3
  --member-clusters="${K8S_CLUSTER_0_CONTEXT_NAME},${K8S_CLUSTER_1_CONTEXT_NAME},${K8S_CLUSTER_2_CONTEXT_NAME}" \
4
  --member-cluster-namespace="${NAMESPACE}" \
5
  --central-cluster-namespace="${OPERATOR_NAMESPACE}" \
6
  --create-service-account-secrets \
7
  --install-database-roles=true \
8
  --image-pull-secrets=image-registries-secret

Install the Kubernetes Operator using the Helm chart.

1. Add and update the MongoDB Helm repository. Verify that the local Helm cache refers to the correct Kubernetes Operator version:

   1
   helm repo add mongodb https://mongodb.github.io/helm-charts
   2
   helm repo update mongodb
   3
   helm search repo "${OFFICIAL_OPERATOR_HELM_CHART}"

   VIEW OUTPUT

2. Install the Kubernetes Operator into the $OPERATOR_NAMESPACE, configured to watch $NAMESPACE and to manage three member Kubernetes clusters. At this point in the procedure, ServiceAccounts and roles are already deployed by the kubectl mongodb plugin. Therefore, the following scripts skip configuring them and set operator.createOperatorServiceAccount=false and operator.createResourcesServiceAccountsAndRoles=false. The scripts specify the multiCluster.clusters setting to instruct the Helm chart to deploy the Kubernetes Operator in multi-cluster mode.

   1
   helm upgrade --install \
   2
     --debug \
   3
     --kube-context "${K8S_CLUSTER_0_CONTEXT_NAME}" \
   4
     mongodb-enterprise-operator-multi-cluster \
   5
     "${OPERATOR_HELM_CHART}" \
   6
     --namespace="${OPERATOR_NAMESPACE}" \
   7
     --set namespace="${OPERATOR_NAMESPACE}" \
   8
     --set operator.namespace="${OPERATOR_NAMESPACE}" \
   9
     --set operator.watchNamespace="${NAMESPACE}" \
   10
     --set operator.name=mongodb-enterprise-operator-multi-cluster \
   11
     --set operator.createOperatorServiceAccount=false \
   12
     --set operator.createResourcesServiceAccountsAndRoles=false \
   13
     --set "multiCluster.clusters={${K8S_CLUSTER_0_CONTEXT_NAME},${K8S_CLUSTER_1_CONTEXT_NAME},${K8S_CLUSTER_2_CONTEXT_NAME}}" \
   14
     --set "${OPERATOR_ADDITIONAL_HELM_VALUES:-"dummy=value"}"

   VIEW OUTPUT

3. Check the Kubernetes Operator deployment:

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" rollout status deployment/mongodb-enterprise-operator-multi-cluster
   2
   echo "Operator deployment in ${OPERATOR_NAMESPACE} namespace"
   3
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" get deployments
   4
   echo; echo "Operator pod in ${OPERATOR_NAMESPACE} namespace"
   5
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" get pods

   VIEW OUTPUT

Prepare TLS certificates.

In this step, you enable TLS for the Application Database and the Ops Manager Application. If you don't want to use TLS, remove the following fields from the MongoDBOpsManager resources:

• spec.security.certsSecretPrefix

• spec.security.tls.ca

• spec.applicationDatabase.security.certsSecretPrefix

• spec.applicationDatabase.security.tls.ca

1. Optional. Generate keys and certificates:

   Use the openssl command line tool to generate self-signed CAs and certificates for testing purposes.

   1
   mkdir certs || true
   2
   3
   cat <<EOF >certs/ca.cnf
   4
   [ req ]
   5
   default_bits       = 2048
   6
   prompt             = no
   7
   default_md         = sha256
   8
   distinguished_name = dn
   9
   x509_extensions    = v3_ca
   10
   11
   [ dn ]
   12
   C=US
   13
   ST=New York
   14
   L=New York
   15
   O=Example Company
   16
   OU=IT Department
   17
   CN=exampleCA
   18
   19
   [ v3_ca ]
   20
   basicConstraints = CA:TRUE
   21
   keyUsage = critical, keyCertSign, cRLSign
   22
   subjectKeyIdentifier = hash
   23
   authorityKeyIdentifier = keyid:always,issuer
   24
   EOF
   25
   26
   cat <<EOF >certs/om.cnf
   27
   [ req ]
   28
   default_bits       = 2048
   29
   prompt             = no
   30
   default_md         = sha256
   31
   distinguished_name = dn
   32
   req_extensions     = req_ext
   33
   34
   [ dn ]
   35
   C=US
   36
   ST=New York
   37
   L=New York
   38
   O=Example Company
   39
   OU=IT Department
   40
   CN=${OPS_MANAGER_EXTERNAL_DOMAIN}
   41
   42
   [ req_ext ]
   43
   subjectAltName = @alt_names
   44
   keyUsage = critical, digitalSignature, keyEncipherment
   45
   extendedKeyUsage = serverAuth, clientAuth
   46
   47
   [ alt_names ]
   48
   DNS.1 = ${OPS_MANAGER_EXTERNAL_DOMAIN}
   49
   DNS.2 = om-svc.${NAMESPACE}.svc.cluster.local
   50
   EOF
   51
   52
   cat <<EOF >certs/appdb.cnf
   53
   [ req ]
   54
   default_bits       = 2048
   55
   prompt             = no
   56
   default_md         = sha256
   57
   distinguished_name = dn
   58
   req_extensions     = req_ext
   59
   60
   [ dn ]
   61
   C=US
   62
   ST=New York
   63
   L=New York
   64
   O=Example Company
   65
   OU=IT Department
   66
   CN=AppDB
   67
   68
   [ req_ext ]
   69
   subjectAltName = @alt_names
   70
   keyUsage = critical, digitalSignature, keyEncipherment
   71
   extendedKeyUsage = serverAuth, clientAuth
   72
   73
   [ alt_names ]
   74
   # multi-cluster mongod hostnames from service for each pod
   75
   DNS.1 = *.${NAMESPACE}.svc.cluster.local
   76
   # single-cluster mongod hostnames from headless service
   77
   DNS.2 = *.om-db-svc.${NAMESPACE}.svc.cluster.local
   78
   EOF
   79
   80
   # generate CA keypair and certificate
   81
   openssl genrsa -out certs/ca.key 2048
   82
   openssl req -x509 -new -nodes -key certs/ca.key -days 1024 -out certs/ca.crt -config certs/ca.cnf
   83
   84
   # generate OpsManager's keypair and certificate
   85
   openssl genrsa -out certs/om.key 2048
   86
   openssl req -new -key certs/om.key -out certs/om.csr -config certs/om.cnf
   87
   88
   # generate AppDB's keypair and certificate
   89
   openssl genrsa -out certs/appdb.key 2048
   90
   openssl req -new -key certs/appdb.key -out certs/appdb.csr -config certs/appdb.cnf
   91
   92
   # generate certificates signed by CA for OpsManager and AppDB
   93
   openssl x509 -req -in certs/om.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/om.crt -days 365 -sha256 -extfile certs/om.cnf -extensions req_ext
   94
   openssl x509 -req -in certs/appdb.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/appdb.crt -days 365 -sha256 -extfile certs/appdb.cnf -extensions req_ext

2. Create secrets with TLS keys:

   If you prefer to use your own keys and certificates, skip the previous generation step and put the keys and certificates into the following files:

   • certs/ca.crt - CA certificates. These are not necessary when using trusted certificates.

   • certs/appdb.key - private key for the Application Database.

   • certs/appdb.crt - certificate for the Application Database.

   • certs/om.key - private key for Ops Manager.

   • certs/om.crt - certificate for Ops Manager.

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret tls cert-prefix-om-cert \
   2
     --cert=certs/om.crt \
   3
     --key=certs/om.key
   4
   5
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret tls cert-prefix-om-db-cert \
   6
     --cert=certs/appdb.crt \
   7
     --key=certs/appdb.key
   8
   9
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create configmap om-cert-ca --from-file="mms-ca.crt=certs/ca.crt"
   10
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create configmap appdb-cert-ca --from-file="ca-pem=certs/ca.crt"

Install Ops Manager.

At this point, you have prepared the environment and the Kubernetes Operator to deploy the Ops Manager resource.

1. Create the necessary credentials for the Ops Manager admin user that the Kubernetes Operator will create after deploying the Ops Manager Application instance:

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" --namespace "${NAMESPACE}" create secret generic om-admin-user-credentials \
   2
     --from-literal=Username="admin" \
   3
     --from-literal=Password="Passw0rd@" \
   4
     --from-literal=FirstName="Jane" \
   5
     --from-literal=LastName="Doe"

2. Deploy the simplest MongoDBOpsManager custom resource possible (with TLS enabled) on a single member cluster, which is also known as the operator cluster.

   This deployment is almost the same as for the single-cluster mode, but with spec.topology and spec.applicationDatabase.topology set to MultiCluster.

   Deploying this way shows that a single Kubernetes cluster deployment is a special case of a multi-Kubernetes cluster deployment on a single Kubernetes member cluster. You can start deploying the Ops Manager Application and the Application Database on as many Kubernetes clusters as necessary from the beginning, and don't have to start with the deployment with only a single member Kubernetes cluster.

   At this point, you have prepared the Ops Manager deployment to span more than one Kubernetes cluster, which you will do later in this procedure.

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: mongodb.com/v1
   3
   kind: MongoDBOpsManager
   4
   metadata:
   5
     name: om
   6
   spec:
   7
     topology: MultiCluster
   8
     version: "${OPS_MANAGER_VERSION}"
   9
     adminCredentials: om-admin-user-credentials
   10
     security:
   11
       certsSecretPrefix: cert-prefix
   12
       tls:
   13
         ca: om-cert-ca
   14
     clusterSpecList:
   15
       - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   16
         members: 1
   17
     applicationDatabase:
   18
       version: "${APPDB_VERSION}"
   19
       topology: MultiCluster
   20
       security:
   21
         certsSecretPrefix: cert-prefix
   22
         tls:
   23
           ca: appdb-cert-ca
   24
       clusterSpecList:
   25
         - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   26
           members: 3
   27
     backup:
   28
       enabled: false
   29
   EOF

3. Wait for the Kubernetes Operator to pick up the work and reach the status.applicationDatabase.phase=Pending state. Wait for both the Application Database and Ops Manager deployments to complete.

   1
   echo "Waiting for Application Database to reach Pending phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s

   VIEW OUTPUT

4. Deploy Ops Manager. The Kubernetes Operator deploys Ops Manager by performing the following steps. It:

   • Deploys the Application Database's replica set nodes and waits for the MongoDB processes in the replica set to start running.

   • Deploys the Ops Manager Application instance with the Application Database's connection string and waits for it to become ready.

   • Adds the Monitoring MongoDB Agent containers to each Application Database's Pod.

   • Waits for both the Ops Manager Application and the Application Database Pods to start running.

   1
   echo "Waiting for Application Database to reach Running phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
   3
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   4
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s
   5
   echo; echo "Waiting for Application Database to reach Pending phase (enabling monitoring)..."
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
   7
   echo "Waiting for Application Database to reach Running phase..."
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
   9
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   10
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s
   11
   echo; echo "MongoDBOpsManager resource"
   12
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
   13
   echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   14
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   15
   echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   16
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

   VIEW OUTPUT

   Now that you have deployed a single-member cluster in a multi-cluster mode, you can reconfigure this deployment to span more than one Kubernetes cluster.

5. On the second member cluster, deploy two additional Application Database replica set members and one additional instance of the Ops Manager Application:

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: mongodb.com/v1
   3
   kind: MongoDBOpsManager
   4
   metadata:
   5
     name: om
   6
   spec:
   7
     topology: MultiCluster
   8
     version: "${OPS_MANAGER_VERSION}"
   9
     adminCredentials: om-admin-user-credentials
   10
     security:
   11
       certsSecretPrefix: cert-prefix
   12
       tls:
   13
         ca: om-cert-ca
   14
     clusterSpecList:
   15
       - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   16
         members: 1
   17
       - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   18
         members: 1
   19
     applicationDatabase:
   20
       version: "${APPDB_VERSION}"
   21
       topology: MultiCluster
   22
       security:
   23
         certsSecretPrefix: cert-prefix
   24
         tls:
   25
           ca: appdb-cert-ca
   26
       clusterSpecList:
   27
         - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   28
           members: 3
   29
         - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   30
           members: 2
   31
     backup:
   32
       enabled: false
   33
   EOF

6. Wait for the Kubernetes Operator to pick up the work (pending phase):

   1
   echo "Waiting for Application Database to reach Pending phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s

   VIEW OUTPUT

7. Wait for the Kubernetes Operator to finish deploying all components:

   1
   echo "Waiting for Application Database to reach Running phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s
   3
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   4
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s
   5
   echo; echo "MongoDBOpsManager resource"
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
   7
   echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   9
   echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   10
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

   VIEW OUTPUT

Enable backup.

In a multi-Kubernetes cluster deployment of the Ops Manager Application, you can configure only S3-based backup storage. This procedure refers to S3_* defined in env_variables.sh.

1. Optional. Install the MinIO Operator.

   This procedure deploys S3-compatible storage for your backups using the MinIO Operator. You can skip this step if you have AWS S3 or other S3-compatible buckets available. Adjust the S3_* variables accordingly in env_variables.sh in this case.

   1
   kubectl kustomize "github.com/minio/operator/resources/?timeout=120&ref=v5.0.12" | \
   2
     kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
   3
   4
   kubectl kustomize "github.com/minio/operator/examples/kustomization/tenant-tiny?timeout=120&ref=v5.0.12" | \
   5
     kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
   6
   7
   # add two buckets to the tenant config
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "tenant-tiny" patch tenant/myminio \
   9
     --type='json' \
   10
     -p="[{\"op\": \"add\", \"path\": \"/spec/buckets\", \"value\": [{\"name\": \"${S3_OPLOG_BUCKET_NAME}\"}, {\"name\": \"${S3_SNAPSHOT_BUCKET_NAME}\"}]}]"

2. Before you configure and enable backup, create secrets:

   • s3-access-secret - contains S3 credentials.

   • s3-ca-cert - contains a CA certificate that issued the bucket's server certificate. In the case of the sample MinIO deployment used in this procedure, the default Kubernetes Root CA certificate is used to sign the certificate. Because it's not a publicly trusted CA certificate, you must provide it so that Ops Manager can trust the connection.

   If you use publicly trusted certificates, you may skip this step and remove the values from the spec.backup.s3Stores.customCertificateSecretRefs and spec.backup.s3OpLogStores.customCertificateSecretRefs settings.

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic s3-access-secret \
   2
     --from-literal=accessKey="${S3_ACCESS_KEY}" \
   3
     --from-literal=secretKey="${S3_SECRET_KEY}"
   4
   5
   # minio TLS secrets are signed with the default k8s root CA
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic s3-ca-cert \
   7
     --from-literal=ca.crt="$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n kube-system get configmap kube-root-ca.crt -o jsonpath="{.data.ca\.crt}")"

Re-deploy Ops Manager with backup enabled.

1. The Kubernetes Operator can configure and deploy all components, the Ops Manager Application, the Backup Daemon instances, and the Application Database's replica set nodes in any combination on any member clusters for which you configure the Kubernetes Operator.

   To illustrate the flexibility of the multi-Kubernetes cluster deployment configuration, deploy only one Backup Daemon instance on the third member cluster and specify zero Backup Daemon members for the first and second clusters.

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: mongodb.com/v1
   3
   kind: MongoDBOpsManager
   4
   metadata:
   5
     name: om
   6
   spec:
   7
     topology: MultiCluster
   8
     version: "${OPS_MANAGER_VERSION}"
   9
     adminCredentials: om-admin-user-credentials
   10
     security:
   11
       certsSecretPrefix: cert-prefix
   12
       tls:
   13
         ca: om-cert-ca
   14
     clusterSpecList:
   15
       - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   16
         members: 1
   17
         backup:
   18
           members: 0
   19
       - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   20
         members: 1
   21
         backup:
   22
           members: 0
   23
       - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}"
   24
         members: 0
   25
         backup:
   26
           members: 1
   27
     configuration: # to avoid configuration wizard on first login
   28
         mms.adminEmailAddr: email@example.com
   29
         mms.fromEmailAddr: email@example.com
   30
         mms.ignoreInitialUiSetup: "true"
   31
         mms.mail.hostname: smtp@example.com
   32
         mms.mail.port: "465"
   33
         mms.mail.ssl: "true"
   34
         mms.mail.transport: smtp
   35
         mms.minimumTLSVersion: TLSv1.2
   36
         mms.replyToEmailAddr: email@example.com
   37
     applicationDatabase:
   38
       version: "${APPDB_VERSION}"
   39
       topology: MultiCluster
   40
       security:
   41
         certsSecretPrefix: cert-prefix
   42
         tls:
   43
           ca: appdb-cert-ca
   44
       clusterSpecList:
   45
         - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   46
           members: 3
   47
         - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   48
           members: 2
   49
     backup:
   50
       enabled: true
   51
       s3Stores:
   52
         - name: my-s3-block-store
   53
           s3SecretRef:
   54
             name: "s3-access-secret"
   55
           pathStyleAccessEnabled: true
   56
           s3BucketEndpoint: "${S3_ENDPOINT}"
   57
           s3BucketName: "${S3_SNAPSHOT_BUCKET_NAME}"
   58
           customCertificateSecretRefs:
   59
             - name: s3-ca-cert
   60
               key: ca.crt
   61
       s3OpLogStores:
   62
         - name: my-s3-oplog-store
   63
           s3SecretRef:
   64
             name: "s3-access-secret"
   65
           s3BucketEndpoint: "${S3_ENDPOINT}"
   66
           s3BucketName: "${S3_OPLOG_BUCKET_NAME}"
   67
           pathStyleAccessEnabled: true
   68
           customCertificateSecretRefs:
   69
             - name: s3-ca-cert
   70
               key: ca.crt
   71
   EOF

2. Wait until the Kubernetes Operator finishes its configuration:

   1
   echo; echo "Waiting for Backup to reach Running phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.backup.phase}'=Running opsmanager/om --timeout=1200s
   3
   echo "Waiting for Application Database to reach Running phase..."
   4
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s
   5
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s
   7
   echo; echo "MongoDBOpsManager resource"
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
   9
   echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   10
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   11
   echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   12
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   13
   echo; echo "Pods running in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"
   14
   kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

   VIEW OUTPUT

Optional. Delete the GKE (Google Kubernetes Engine) clusters and all their associated resources (VMs).

Run the following script to delete the GKE clusters and clean up your environment.

1
yes | gcloud container clusters delete "${K8S_CLUSTER_0}" --zone="${K8S_CLUSTER_0_ZONE}" &
2
yes | gcloud container clusters delete "${K8S_CLUSTER_1}" --zone="${K8S_CLUSTER_1_ZONE}" &
3
yes | gcloud container clusters delete "${K8S_CLUSTER_2}" --zone="${K8S_CLUSTER_2_ZONE}" &
4
wait