• Deploy MongoDB Resources on Multiple Kubernetes Clusters >
• Multi-Kubernetes-Cluster Quick Start

Multi-Kubernetes-Cluster Quick Start

Use the quick start to deploy a MongoDB replica set across three Kubernetes member clusters, using GKE (Google Kubernetes Engine) and Istio service mesh.

Before you begin:

• Learn about multi-Kubernetes-cluster deployments
• Review the list of multi-Kubernetes-cluster services and tools
• Complete the prerequisites

Note

The following procedures scope your multi-Kubernetes-cluster deployment to a single namespace named mongodb. You can configure your multi-Kubernetes-cluster deployment to watch resources in multiple namespaces or all namespaces.

Prefer to Learn by Watching?

Follow along with this video tutorial walk-through that demonstrates how to create a multi-Kubernetes-cluster deployment.

Duration: 12 Minutes

Deploying a MongoDB Replica Set across Multiple Kubernetes Clusters

Deploy a MongoDBMultiCluster Resource

Select the appropriate tab based on whether you want to encrypt replica set connections in your multi-Kubernetes-cluster deployments using TLS certificates.

• With TLS-Encrypted Connections
• With Non-Encrypted Connections

You can use the following procedures in this TLS-Encrypted Connections tab:

• Deploy a MongoDBMultiCluster resource
• Renew TLS Certificates for a MongoDBMultiCluster resource

These procedures establish TLS-encrypted connections between MongoDB hosts in a replica set, and between client applications and MongoDB deployments.

Before you begin, you must have valid certificates for TLS encryption.

1

Create the secret for the TLS certificate of your MongoDBMultiCluster resource.

Run the kubectl command to create a new secret that stores the MongoDBMultiCluster resource certificate:

copy

kubectl --context $MDB_CENTRAL_CLUSTER_FULL_NAME \
  --namespace=<metadata.namespace> \
  create secret tls <prefix>-<metadata.name>-cert \
  --cert=<resource-tls-cert> \
  --key=<resource-tls-key>

Note

You must prefix your secrets with <prefix>-<metadata.name>.

Example

If you call your deployment my-deployment and you set the prefix to mdb, you must name the TLS secret for the client TLS communications mdb-my-deployment-cert. Also, you must name the TLS secret for internal cluster authentication (if enabled) mdb-my-deployment-clusterfile.

2

Create the ConfigMap to link your CA with your MongoDBMultiCluster resource.

Run the kubectl command to link your CA to your MongoDBMultiCluster resource. Specify the CA certificate file that you must always name ca-pem for the MongoDBMultiCluster resource:

copy

kubectl --context $MDB_CENTRAL_CLUSTER_FULL_NAME \
  --namespace=<metadata.namespace> \
  create configmap custom-ca -from-file=ca-pem=<your-custom-ca-file>

3

Run the kubectl mongodb plugin.

By default, the Kubernetes Operator is scoped to the mongodb namespace. When you run the following command, the kubectl mongodb plugin:

• Creates one central cluster, three member clusters, and a namespace labeled mongodb in each of the clusters.
• Creates a default ConfigMap with the hard-coded name mongodb-enterprise-operator-member-list that contains all the member clusters. You can’t change the ConfigMap’s name.
• Creates the service accounts and Roles required for running database workloads in the member clusters.

Run the kubectl mongodb plugin:

copy

kubectl mongodb multicluster setup \
  --central-cluster="${MDB_CENTRAL_CLUSTER_FULL_NAME}" \
  --member-clusters="${MDB_CLUSTER_1_FULL_NAME},${MDB_CLUSTER_2_FULL_NAME},${MDB_CLUSTER_3_FULL_NAME}" \
  --member-cluster-namespace="mongodb" \
  --central-cluster-namespace="mongodb" \
  --create-service-account-secrets \
  --install-database-roles=true

4

Optional: Set the Istio injection webhook in each member cluster.

If you’re using Istio, run the following command on the central cluster, specifying the context for each of the member clusters in the deployment. These commands add the istio-injection=enabled label to the mongodb namespace on each member cluster. This label configures Istio’s injection webhook which enables adding a sidecar to any Pods that you create in this namespace. To learn more, see Automatic sidecar injection in the Istio documentation.

copy

kubectl label \
  --context=$MDB_CLUSTER_1_FULL_NAME \
  namespace mongodb \
  istio-injection=enabled

copy

kubectl label \
  --context=$MDB_CLUSTER_2_FULL_NAME \
  namespace mongodb \
  istio-injection=enabled

copy

kubectl label \
  --context=$MDB_CLUSTER_3_FULL_NAME \
  namespace mongodb \
  istio-injection=enabled

5

Configure kubectl to use the central cluster’s namespace.

If you have not done so already, run the following commands to run all kubectl commands on the central cluster in the default namespace.

copy

kubectl config use-context $MDB_CENTRAL_CLUSTER_FULL_NAME
kubectl config set-context $(kubectl config current-context) \
  --namespace=mongodb

6

Deploy the MongoDB Enterprise Kubernetes Operator in the central cluster.

Deploy the MongoDB Enterprise Kubernetes Operator in the central cluster in the mongodb namespace with Helm or kubectl.

• Helm
• kubectl

1. Add the MongoDB Helm Charts for Kubernetes repository to Helm.

   copy

   helm repo add mongodb https://mongodb.github.io/helm-charts
   

2. Use the MongoDB Helm Charts for Kubernetes to deploy the Kubernetes Operator.

   copy

   helm upgrade \
     --install \
       mongodb-enterprise-operator-multi-cluster \
       mongodb/enterprise-operator \
         --namespace mongodb \
         --set namespace=mongodb \
         --version <mongodb-kubernetes-operator-version> \
         --set operator.name=mongodb-enterprise-operator-multi-cluster \
         --set operator.createOperatorServiceAccount=false \
         --set "multiCluster.clusters={$MDB_CLUSTER_1_FULL_NAME,$MDB_CLUSTER_2_FULL_NAME,$MDB_CLUSTER_3_FULL_NAME}" \
         --set multiCluster.performFailover=false
   

1. Apply the Kubernetes Operator custom resources.

   copy

   kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-enterprise-kubernetes/master/crds.yaml
   

2. Download the Kubernetes Operator YAML template.

   copy

   curl https://raw.githubusercontent.com/mongodb/mongodb-enterprise-kubernetes/master/mongodb-enterprise-multi-cluster.yaml -o operator.yaml
   

3. Optional: Customize the Kubernetes Operator YAML template.

   To learn about optional Kubernetes Operator installation settings, see MongoDB Enterprise Kubernetes Operator kubectl and oc Installation Settings.

4. Apply the Kubernetes Operator YAML file.

   copy

   kubectl apply -f operator.yaml
   

5. Verify that the Kubernetes Operator is deployed.

   To verify that the Kubernetes Operator installed correctly, run the following command and verify the output:

   • Using kubectl
   • Using oc

   copy

   kubectl describe deployments mongodb-enterprise-operator -n <metadata.namespace>
   

   copy

   oc describe deployments mongodb-enterprise-operator -n <metadata.namespace>
   

   By default, deployments exist in the mongodb namespace. If the following error message appears, ensure you use the correct namespace:

   copy

   Error from server (NotFound): deployments.apps "mongodb-enterprise-operator" not found
   

   To troubleshoot your Kubernetes Operator, see Review Logs from the Kubernetes Operator and other troubleshooting topics.

   Important

   If you need to remove the Kubernetes Operator or the namespace, you first must remove MongoDB resources.

7

Create a secret to use with Ops Manager and create a ConfigMap.

1. Create a secret so that the Kubernetes Operator can create and update objects in your Ops Manager project. To learn more, see Create Credentials for the Kubernetes Operator.
2. Create a ConfigMap to link the Kubernetes Operator to your Ops Manager project. To learn more, see Create One Project using a ConfigMap.

8

Deploy Ops Manager on the central cluster and connect to Ops Manager.

Review the multi-Kubernetes-cluster deployment exceptions and follow the linked procedures in Deploy an Ops Manager Resource, deploy the Application Database, and Connect to Ops Manager.

9

Deploy the MongoDBMultiCluster resource.

Set spec.credentials, spec.opsManagerconfigMapRef.name, and security settings and deploy the MongoDBMultiCluster resource. In the following code sample, duplicateServiceObjects is set to true to enable DNS proxying in Istio.

Note

To enable the cross-cluster DNS resolution by the Istio service mesh, this tutorial creates service objects with a single ClusterIP address per each Kubernetes Pod.

copy

kubectl apply -f - <<EOF
apiVersion: mongodb.com/v1
kind: MongoDBMultiCluster
metadata:
 name: multi-replica-set
spec:
 version: 4.4.0-ent
 type: ReplicaSet
 persistent: false
 duplicateServiceObjects: true
 credentials: my-credentials
 opsManager:
   configMapRef:
     name: my-project
 security:
   certsSecretPrefix: <prefix>
   tls:
     ca: custom-ca
 clusterSpecList:
   - clusterName: ${MDB_CLUSTER_1_FULL_NAME}
     members: 3
   - clusterName: ${MDB_CLUSTER_2_FULL_NAME}
     members: 2
   - clusterName: ${MDB_CLUSTER_3_FULL_NAME}
     members: 3
EOF

The Kubernetes Operator copies the ConfigMap with the CA that you created in previous steps to each member cluster, generates a concatenated PEM secret, and distributes it to the member clusters.

10

Verify that the MongoDBMultiCluster resources are running.

1. For member clusters, run the following commands to verify that the MongoDB Pods are in the running state:

   copy

   kubectl get pods \
    --context=$MDB_CLUSTER_1_FULL_NAME \
    --namespace mongodb
   

   copy

   kubectl get pods \
    --context=$MDB_CLUSTER_2_FULL_NAME \
    --namespace mongodb
   

   copy

   kubectl get pods \
    --context=$MDB_CLUSTER_3_FULL_NAME \
    --namespace mongodb
   

2. In the central cluster, run the following command to verify that the MongoDBMultiCluster resource is in the running state:

   copy

   kubectl --context=$MDB_CENTRAL_CLUSTER_FULL_NAME \
     --namespace mongodb \
     get mdbmc multi-replica-set -o yaml -w
   

Renew TLS Certificates for a MongoDBMultiCluster Resource

Renew your TLS certificates periodically using the following procedure.

1

Renew the secret for a MongoDBMultiCluster resource.

Run this kubectl command to renew an existing secret that stores the certificates for the MongoDBMultiCluster resource:

copy

kubectl --context $MDB_CENTRAL_CLUSTER_FULL_NAME \
--namespace=<metadata.namespace> \
create secret tls <prefix>-<metadata.name>-cert \
--cert=<resource-tls-cert> \
--key=<resource-tls-key> \
--dry-run=client \
-o yaml |
kubectl apply -f -

This procedure doesn’t encrypt connections between MongoDB hosts in a replica set, and between client applications and MongoDB deployments.

1

Run the kubectl mongodb plugin.

By default, the Kubernetes Operator is scoped to the mongodb namespace. When you run the following command, the kubectl mongodb plugin:

• Creates one central cluster, three member clusters, and a namespace labeled mongodb in each of the clusters.
• Creates a default ConfigMap with the hard-coded name mongodb-enterprise-operator-member-list that contains all the member clusters. You can’t change the ConfigMap’s name.
• Creates the service accounts and Roles required for running database workloads in the member clusters.

Run the kubectl mongodb plugin:

copy

kubectl mongodb multicluster setup \
  --central-cluster="${MDB_CENTRAL_CLUSTER_FULL_NAME}" \
  --member-clusters="${MDB_CLUSTER_1_FULL_NAME},${MDB_CLUSTER_2_FULL_NAME},${MDB_CLUSTER_3_FULL_NAME}" \
  --member-cluster-namespace="mongodb" \
  --central-cluster-namespace="mongodb" \
  --create-service-account-secrets \
  --install-database-roles=true

2

Optional: Set the Istio injection webhook in each member cluster.

If you’re using Istio, run the following command on the central cluster, specifying the context for each of the member clusters in the deployment. These commands add the istio-injection=enabled label to the mongodb namespace on each member cluster. This label configures Istio’s injection webhook which enables adding a sidecar to any Pods that you create in this namespace. To learn more, see Automatic sidecar injection in the Istio documentation.

copy

kubectl label \
  --context=$MDB_CLUSTER_1_FULL_NAME \
  namespace mongodb \
  istio-injection=enabled

copy

kubectl label \
  --context=$MDB_CLUSTER_2_FULL_NAME \
  namespace mongodb \
  istio-injection=enabled

copy

kubectl label \
  --context=$MDB_CLUSTER_3_FULL_NAME \
  namespace mongodb \
  istio-injection=enabled

3

Configure kubectl to use the central cluster’s namespace.

If you have not done so already, run the following commands to run all kubectl commands on the central cluster in the default namespace.

copy

kubectl config use-context $MDB_CENTRAL_CLUSTER_FULL_NAME
kubectl config set-context $(kubectl config current-context) \
  --namespace=mongodb

4

Deploy the MongoDB Enterprise Kubernetes Operator in the central cluster.

Deploy the MongoDB Enterprise Kubernetes Operator in the central cluster in the mongodb namespace with Helm or kubectl.

• Helm
• kubectl

1. Add the MongoDB Helm Charts for Kubernetes repository to Helm.

   copy

   helm repo add mongodb https://mongodb.github.io/helm-charts
   

2. Use the MongoDB Helm Charts for Kubernetes to deploy the Kubernetes Operator.

   copy

   helm upgrade \
     --install \
       mongodb-enterprise-operator-multi-cluster \
       mongodb/enterprise-operator \
         --namespace mongodb \
         --set namespace=mongodb \
         --version <mongodb-kubernetes-operator-version> \
         --set operator.name=mongodb-enterprise-operator-multi-cluster \
         --set operator.createOperatorServiceAccount=false \
         --set "multiCluster.clusters={$MDB_CLUSTER_1_FULL_NAME,$MDB_CLUSTER_2_FULL_NAME,$MDB_CLUSTER_3_FULL_NAME}" \
         --set multiCluster.performFailover=false
   

1. Apply the Kubernetes Operator custom resources.

   copy

   kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-enterprise-kubernetes/master/crds.yaml
   

2. Download the Kubernetes Operator YAML template.

   copy

   curl https://raw.githubusercontent.com/mongodb/mongodb-enterprise-kubernetes/master/mongodb-enterprise-multi-cluster.yaml -o operator.yaml
   

3. Optional: Customize the Kubernetes Operator YAML template.

   To learn about optional Kubernetes Operator installation settings, see MongoDB Enterprise Kubernetes Operator kubectl and oc Installation Settings.

4. Apply the Kubernetes Operator YAML file.

   copy

   kubectl apply -f operator.yaml
   

5. Verify that the Kubernetes Operator is deployed.

   To verify that the Kubernetes Operator installed correctly, run the following command and verify the output:

   • Using kubectl
   • Using oc

   copy

   kubectl describe deployments mongodb-enterprise-operator -n <metadata.namespace>
   

   copy

   oc describe deployments mongodb-enterprise-operator -n <metadata.namespace>
   

   By default, deployments exist in the mongodb namespace. If the following error message appears, ensure you use the correct namespace:

   copy

   Error from server (NotFound): deployments.apps "mongodb-enterprise-operator" not found
   

   To troubleshoot your Kubernetes Operator, see Review Logs from the Kubernetes Operator and other troubleshooting topics.

   Important

   If you need to remove the Kubernetes Operator or the namespace, you first must remove MongoDB resources.

5

Create a secret to use with Ops Manager and create a ConfigMap.

1. Create a secret so that the Kubernetes Operator can create and update objects in your Ops Manager project. To learn more, see Create Credentials for the Kubernetes Operator.
2. Create a ConfigMap to link the Kubernetes Operator to your Ops Manager project. To learn more, see Create One Project using a ConfigMap.

6

Deploy Ops Manager on the central cluster and connect to Ops Manager.

Review the multi-Kubernetes-cluster deployment exceptions and follow the linked procedures in Deploy an Ops Manager Resource, deploy the Application Database, and Connect to Ops Manager.

7

Deploy the MongoDBMultiCluster resource.

Set spec.credentials, spec.opsManagerconfigMapRef.name, and security settings and deploy the MongoDBMultiCluster resource. In the following code sample, duplicateServiceObjects is set to true to enable DNS proxying in Istio.

Note

To enable the cross-cluster DNS resolution by the Istio service mesh, this tutorial creates service objects with a single ClusterIP address per each Kubernetes Pod.

copy

kubectl apply -f - <<EOF
apiVersion: mongodb.com/v1
kind: MongoDBMultiCluster
metadata:
 name: multi-replica-set
spec:
 version: 4.4.0-ent
 type: ReplicaSet
 persistent: false
 duplicateServiceObjects: true
 credentials: my-credentials
 opsManager:
   configMapRef:
     name: my-project
 security:
   certsSecretPrefix: <prefix>
   tls:
     ca: custom-ca
 clusterSpecList:
   - clusterName: ${MDB_CLUSTER_1_FULL_NAME}
     members: 3
   - clusterName: ${MDB_CLUSTER_2_FULL_NAME}
     members: 2
   - clusterName: ${MDB_CLUSTER_3_FULL_NAME}
     members: 3
EOF

The Kubernetes Operator copies the ConfigMap with the CA that you created in previous steps to each member cluster, generates a concatenated PEM secret, and distributes it to the member clusters.

8

Verify that the MongoDBMultiCluster resources are running.

1. For member clusters, run the following commands to verify that the MongoDB Pods are in the running state:

   copy

   kubectl get pods \
    --context=$MDB_CLUSTER_1_FULL_NAME \
    --namespace mongodb
   

   copy

   kubectl get pods \
    --context=$MDB_CLUSTER_2_FULL_NAME \
    --namespace mongodb
   

   copy

   kubectl get pods \
    --context=$MDB_CLUSTER_3_FULL_NAME \
    --namespace mongodb
   

2. In the central cluster, run the following command to verify that the MongoDBMultiCluster resource is in the running state:

   copy

   kubectl --context=$MDB_CENTRAL_CLUSTER_FULL_NAME \
     --namespace mongodb \
     get mdbmc multi-replica-set -o yaml -w
   

Next Steps

After deploying your MongoDB replica set across three Kubernetes member clusters, you can add a database user so you can connect to your MongoDB database. See Manage Database Users.

←   Prerequisites Deploy Replica Sets in a Multi-Kubernetes Cluster  →

© MongoDB, Inc 2008-present. MongoDB, Mongo, and the leaf logo are registered trademarks of MongoDB, Inc.