Deploying MongoDB Across Multiple Kubernetes Clusters With MongoDBMulti
Rate this tutorial
This article is part of a three-parts series on deploying MongoDB across multiple Kubernetes clusters using the operators.
- Deploying MongoDB Across Multiple Kubernetes Clusters With MongoDBMulti
With the latest version of the MongoDB Enterprise Kubernetes Operator, you can deploy MongoDB resources across multiple Kubernetes clusters! By running your MongoDB replica set across different clusters, you can ensure that your deployment remains available even in the event of a failure or outage in one of them. The MongoDB Enterprise Kubernetes Operator's Custom Resource Definition (CRD), MongoDBMulti, makes it easy to run MongoDB replica sets across different Kubernetes environments and provides a declarative approach to deploying MongoDB, allowing you to specify the desired state of your deployment and letting the operator handle the details of achieving that state.
⚠️ Support for multi-Kubernetes-cluster deployments of MongoDB is a preview feature and not yet ready for Production use. The content of this article is meant to provide you with a way to experiment with this upcoming feature, but should not be used in production as breaking changes may still occur. Support for this feature during preview is direct with the engineering team and on a best-efforts basis, so please let us know if trying this out at kubernetes-product@mongodb.com. Also feel free to get in touch with any questions, or if this is something that may be of interest once fully released.
Developed by MongoDB, MongoDBMulti Custom Resource allows for the customization of resilience levels based on the needs of the enterprise application.
- Single region (Multi A-Z) consists of one or more Kubernetes clusters where each cluster has nodes deployed in different availability zones in the same region. This type of deployment protects MongoDB instances backing your enterprise applications against zone and Kubernetes cluster failures.
- Multi Region consists of one or more Kubernetes clusters where you deploy each cluster in a different region, and within each region, deploy cluster nodes in different availability zones. This gives your database resilience against the loss of a Kubernetes cluster, a zone, or an entire cloud region.
By leveraging the native capabilities of Kubernetes, the MongoDB Enterprise Kubernetes Operator performs the following tasks to deploy and operate a multi-cluster MongoDB replica set:
- Creates the necessary resources, such as Configmaps, secrets, service objects, and StatefulSet objects, in each member cluster. These resources are in line with the number of replica set members in the MongoDB cluster, ensuring that the cluster is properly configured and able to function.
- Identifies the clusters where the MongoDB replica set should be deployed using the corresponding MongoDBMulti Custom Resource spec. It then deploys the replica set on the identified clusters.
- Watches for the creation of the MongoDBMulti Custom Resource spec in the central cluster.
- Uses a mounted kubeconfig file to communicate with member clusters. This allows the operator to access the necessary information and resources on the member clusters in order to properly manage and configure the MongoDB cluster.
- Watches for events related to the CentralCluster and MemberCluster in order to confirm that the multi-Kubernetes-cluster deployment is in the desired state.
You should start by constructing a central cluster. This central cluster will host the Kubernetes Operator, MongoDBMulti Custom Resource spec, and act as the control plane for the multi-cluster deployment. If you deploy Ops Manager with the Kubernetes Operator, the central cluster may also host Ops Manager.
You will also need a service mesh. I will be using Istio, but any service mesh that provides a fully qualified domain name resolution between pods across clusters should work.
Communication between replica set members happens via the service mesh, which means that your MongoDB replica set doesn't need the central cluster to function. Keep in mind that if the central cluster goes down, you won't be able to use the Kubernetes Operator to modify your deployment until you regain access to this cluster.
Alright, let's get started using the operator and build something! For this tutorial, we will need the following tools:
We need to set up a master Kubernetes cluster to host the MongoDB Enterprise Multi-Cluster Kubernetes Operator and the Ops Manager. You will need to create a GKE Kubernetes cluster by following the instructions in Part 1 of this series. Then, we should install the MongoDB Multi-Cluster Kubernetes Operator in the
mongodb namespace, along with the necessary CRDs. This will allow us to utilize the operator to effectively manage and operate our MongoDB multi cluster replica set. For instructions on how to do this, please refer to the relevant section of Part 1. Additionally, we will need to install the Ops Manager, as outlined in Part 2 of this series.After master cluster creation and configuration, we need three additional GKE clusters, distributed across three different regions:
us-west2, us-central1, and us-east1. Those clusters will host MongoDB replica set members.
The clusters have been created, and we need to obtain the credentials for them.
After successfully creating the Kubernetes master and MongoDB replica set clusters, installing the Ops Manager and all required software on it, we can check them using
[kubectx](https://github.com/ahmetb/kubectx).You should see all your Kubernetes clusters listed here. Make sure that you only have the clusters you just created and remove any other unnecessary clusters using
kubectx -d <cluster_name> for the next script to work.We need to create the required variables:
MASTER for a master Kubernetes cluster, and MDB_1, MDB_2, and MDB_3 for clusters which will host MongoDB replica set members. Important note: These variables should contain the full Kubernetes cluster names.Your clusters are now configured and ready to host the MongoDB Kubernetes Operator.
Install Istio (I'm using v 1.16.1) in a multi-primary mode on different networks, using the install_istio_separate_network script. To learn more about it, see the Multicluster Istio documentation. I have prepared a code that downloads and updates
install_istio_separate_network.sh script variables to currently required ones, such as full K8s cluster names and the version of Istio.Install Istio in a multi-primary mode on different Kubernetes clusters via the following command.
Execute the multi-cluster kubeconfig creator tool. By default, the Kubernetes Operator is scoped to the
mongodb namespace, although it can be installed in a different namespace as well. Navigate to the directory where you cloned the Kubernetes Operator repository in an earlier step, and run the tool. Got to Multi-Cluster CLI documentation to lean more about multi cluster cli.Let's check the configurations we have made so far. I will switch the context to cluster #2.
You should see something like this in your terminal.
We can see
istio-system and mongodb namespaces created by the scriptsand the MongoDB Kubernetes operator service account is ready.
Next, execute the following command on the clusters, specifying the context for each of the member clusters in the deployment. The command adds the label
istio-injection=enabled' to the'mongodb namespace on each member cluster. This label activates Istio's injection webhook, which allows a sidecar to be added to any pods created in this namespace.Now the MongoDB Multi Cluster Kubernetes operator must be installed on the master-operator cluster and be aware of the all Kubernetes clusters which are part of the Multi Cluster. This step will add the multi cluster Kubernetes operator to each of our clusters.
First, switch context to the master cluster.
The
mongodb-operator-multi-cluster operator needs to be made aware of the newly created Kubernetes clusters by updating the operator config through Helm. This procedure was tested with mongodb-operator-multi-cluster version 1.16.3.Check if the MongoDB Enterprise Operator multi cluster pod on the master cluster is running.
It's now time to link all those clusters together using the MongoDB Multi CRD. The Kubernetes API has already been extended with a MongoDB-specific object -
mongodbmulti.You should also review after the installation logs and ensure that there are no issues or errors.
We are almost ready to create a multi cluster MongoDB Kubernetes replica set! We need to configure the required service accounts for each member cluster.
Also, let's generate Ops Manager API keys and add our IP addresses to the Ops Manager access list. Get the Ops Manager (created as described in Part 2) URL. Make sure you switch the context to master.
Log in to Ops Manager, and generate public and private API keys. When you create API keys, don't forget to add your current IP address to API Access List.
To do so, log in to the Ops Manager and go to
ops-manager-db organization.
Click
Access Manager on the left-hand side, and choose Organization Access then choose Create API KEY in the top right corner.
The key must have a name (I use
mongodb-blog) and permissions must be set to Organization Owner .
When you click Next, you will see your
Public Keyand Private Key. Copy those values and save them --- you will not be able to see the private key again. Also, make sure you added your current IP address to the API access list.
Get the public and private keys generated by the API key creator and paste them into the Kubernetes secret.
You also need an
Organization ID. You can see the organization ID by clicking on the gear icon in the top left corner.
Copy the
Organization ID and paste to the Kubernetes config map below.The Ops Manager instance has been configured, and you have everything needed to add the MongoDBMultiCRD to your cluster.
Finally, we can create a MongoDB replica set that is distributed across three Kubernetes clusters in different regions. I have updated the Kubernetes manifest with the full names of the Kubernetes clusters. Let's apply it now!
We should check the operator pod logs. There is a possibility we will have to update Ops Manager API access list by additional IP address. Create a variable with the operator pod.
Check if the operator pod is allowed to access Ops Manager REST API.
If we receive an error output similar to the following, we should add the displayed, additional IP address to the Ops Manager API access list, as we did in the previous step.
Status: 403 (Forbidden), ErrorCode: IP_ADDRESS_NOT_ON_ACCESS_LIST, Detail: IP address 10.206.15.226 is not allowed to access this resource.","MultiReplicaSet":"mongodb/multi-cluster"After a few minutes, we should have our multi cluster ready for use! We can verify this by displaying the 'mongodb multi' object.
We can check that MongoDB replica set is running across different Kubernetes environments in different regions! The operator has performed all necessary configurations and changes to achieve the desired state of the multi cluster. The operator will also monitor our multi cluster and respond in case of any issues.
Output of the loop:
We can also see that the MongoDB Kubernetes Operator has handled the storage part and created a StatefulSet and associated objects.
Output of the loop:
The multi cluster replica set is also visible in the Ops Manager. The Ops Manager will now be responsible for backup, alerting, monitoring, rolling upgrades, and automation.
If something goes wrong, you can start an investigation by checking the MongoDB Kubernetes Operator logs on the master cluster. The most common problem is not including IP addresses in the list of IP addresses for the API key. You can see the error details by finding the name of the operator pod and listing the latest logs.
You can use the kubectl command to view the logs of the database pods. The main container processes continually tail the Automation Agent and MongoDB logs and can be viewed using the following command:
A common technique for troubleshooting issues is to use ssh to connect to one of the containers running MongoDB. Once connected, you can use various Linux tools to view the processes, troubleshoot issues, and even check the MongoDB shell connections (which can be helpful in diagnosing network issues).
MongoDB Enterprise Kubernetes Operator's latest version allows users to deploy MongoDB resources across multiple Kubernetes clusters, improving reliability and reducing downtime. Developed by MongoDB, MongoDBMulti Custom Resource Definition (CRD) makes it easy to run MongoDB replica sets across multiple Kubernetes environments and provides a declarative approach to deploying MongoDB, allowing users to specify the desired state of their deployment and letting the operator handle the details.
In combination with Ops Manager, a multi-region cluster creates a highly available database system with enterprise-class tools for backup, monitoring, alerting, upgrades, and configuration.
As Kubernetes becomes increasingly popular, it's important to start leveraging its capabilities in your organization. Do you want to stay up-to-date on the latest developments in MongoDB on Kubernetes? Be sure to check out MongoDB community forum for the latest discussions and resources.
Rate this tutorial