Multi-Cluster Architecture

Single- and Multi-Cluster Mode

You can control the mode in which you deploy the MongoDB resources with the Kubernetes Operator with the following settings:

• spec.topology — controls the mode for the Ops Manager application.

• spec.applicationDatabase.topology — controls the mode for the Application Database.

If you set both spec.topology and spec.applicationDatabase.topology to MultiCluster, this enables multi-cluster mode for the Ops Manager resource and its Application Database, as shown in the following example:

spec:
  topology: MultiCluster
  applicationDatabase:
    topology: MultiCluster

In multi-cluster mode, you can start with a single member cluster and scale as needed. Specifically:

• Single member cluster start: A deployment can begin with one member cluster.

• Minimum Application Database replica set: A minimum 3-member Application Database replica set can run on a single member cluster, then expand across clusters later.

• Single |application| instance: A single Ops Manager Application instance can run on one cluster, with additional clusters added later.

In single-cluster mode (the default), omit the topology setting or set it to SingleCluster.

Multi-Cluster Deployment Limitations

The following limitations apply to multi-cluster deployments:

• Use Ops Manager versions later than 5.0.7.

• Use only secrets for secret storage. HashiCorp Vault isn't supported.

• For deployments where the Kubernetes Operator doesn't manage both the MongoDBOpsManager and MongoDB resources, you must manually configure KMIP backup encryption settings in Ops Manager.

• Don't add a ServiceMonitor to MongoDBMultiCluster resources. Prometheus integration isn't supported.

• You can't migrate existing single-cluster MongoDB resources to multi-cluster resources.

Differences Between Single and Multi-Cluster Deployments

Multi-Cluster MongoDB Database Resource Deployment Diagrams

You can create multi-cluster MongoDB deployments with or without a service mesh.

In both cases, the Kubernetes Operator:

• Watches for the MongoDBMultiCluster spec on the operator cluster.

• Uses the mounted kubeconfig to communicate with member clusters.

• Creates ConfigMaps, Secrets, Services, and StatefulSets in each member cluster.

• Deploys MongoDB replica set members to the appropriate clusters.

• Watches for operator cluster and member cluster events.

• Reconciles resources to confirm the desired state.

A multi-Kubernetes cluster MongoDB deployment that uses the MongoDB Controllers for Kubernetes Operator consists of one operator cluster and one or more member clusters in Kubernetes:

• The operator cluster has the following role:

  • Hosts the MongoDB Controllers for Kubernetes Operator

  • Acts as the control plane for the multi-Kubernetes cluster MongoDB deployment

  • Hosts the MongoDBMultiCluster resource spec for the MongoDB replica set

  • Hosts Ops Manager, if you deploy Ops Manager with the Kubernetes Operator

  • Can also host members of the MongoDB replica set

  Important

  The central cluster is also known as the operator cluster. References to the central cluster might be renamed to refer to the operator cluster in the future releases.

• Member clusters host the MongoDB replica sets.

With a Service Mesh

The service mesh manages MongoDB node discovery across clusters and handles inter-member communication. The following diagram illustrates a multi-cluster deployment with a service mesh:

click to enlarge

Without a Service Mesh

External domains and DNS zones handle cross-cluster communication. See Enable External Connectivity through External Domains and DNS Zones. The following diagram illustrates a multi-cluster deployment without a service mesh:

click to enlarge

Multi-Cluster Ops Manager Resource Deployment Diagram

The following diagram shows the Ops Manager Application, Application Database, and Backup Daemon deployed on multiple Kubernetes clusters:

click to enlarge

Key elements of a multi-cluster Ops Manager deployment:

1. Operator cluster (for example, Member Cluster 0) stores the kubeconfig secret (mongodb-enterprise-operator-multi-cluster-kubeconfig) and cluster-mapping ConfigMaps used to manage all member clusters.

2. Cluster-mapping ConfigMaps track the relationship between cluster names and indexes:

   • <om_resource_name>-cluster-mapping — maps spec.clusterSpecList entries to cluster indexes.

   • <om_resource_name>-db-cluster-mapping — maps spec.applicationDatabase.clusterSpecList entries to cluster indexes.

   • <om_resource_name>-db-member-spec — records replica count per cluster for disaster recovery.

3. Application StatefulSets are named <om_resource_name>-<cluster_index> on each cluster. The operator creates a ClusterIP service (<om_resource_name>-svc) containing all local Pods, plus an optional LoadBalancer service (<om_resource_name>-svc-ext) for external access.

4. Application Database StatefulSets are named <om_resource_name>-db-<cluster_index>. Per-Pod services (<om_resource_name>-db-<cluster_index>-<pod_index>-svc) enable individual mongod addressability across the service mesh.

5. Backup Daemon StatefulSets are named <om_resource_name>-backup-daemon-<cluster_index>, created if spec.backup.enabled is true.

Multi-Cluster Networking

Load Balancing for Multi-Cluster Ops Manager Deployments

For multi-cluster Ops Manager deployments, each cluster can expose its Ops Manager Application Pods individually using a LoadBalancer-type service. Create this service using spec.externalConnectivity and point an external domain to its external IP address. To learn more about configuring load balancing in multi-cluster deployments, see Service Mesh Requirements for Multi-Cluster Ops Manager Deployments.

Since the Kubernetes Operator doesn't support cross-cluster load balancing natively, you must configure load balancing externally. The following approaches exist for enabling cross-cluster load balancing for Ops Manager deployments:

• External Load Balancer: Configure an external network load balancer (passthrough proxy) for all clusters hosting the Ops Manager Application. The load balancer forwards traffic to each cluster's LoadBalancer service in round-robin fashion. The following diagram illustrates this approach:

click to enlarge

• Service Mesh with Proxy: Use the service mesh's cross-cluster load balancing. Deploy a proxy (such as Nginx or HAProxy) on one cluster, expose it externally, and configure TCP passthrough to <om_resource_name>-svc.<namespace>.svc.cluster.local. The following diagram illustrates this approach:

click to enlarge

Multi-Cluster Performance Considerations for Ops Manager Deployments

The geographical distribution of Application Database and Ops Manager instances can impact the performance of the Ops Manager Application and backup/restore processes. See Performance in Multi-Region Deployments.

To maximize the benefits of multi-cluster resilience and availability, deploy all components in the same geographical area when possible.

Sources of possible performance degradation include:

• Increased network latency between the Ops Manager Application and the primary Application Database node.

• Increased latency between MongoDB database nodes and the Ops Manager Application performing backup jobs.

If you plan geographically distant deployments, contact MongoDB Support for assistance.

Multi-Cluster MongoDB Database Resource Capabilities

The following multi-cluster capabilities use the same procedures as single-cluster deployments:

• Connect with DNS SRV Records: Use the connectionString.standardSrv DNS seed list connection string from the secret that the Kubernetes Operator creates. See Connect to a MongoDB Database Resource from Inside Kubernetes and select the Using the Kubernetes Secret tab.

• Manage Security for Database Users: Use the same LDAP, SCRAM, X.509, and OIDC authentication procedures as single-cluster deployments, with the following exceptions:

  • Multi-cluster procedures apply to replica sets only (sharded clusters aren't supported).

  • In mongodbResourceRef, specify the multi-cluster replica set name.

• Queryable Backups (Single-Cluster |onprem| Only): If Ops Manager is deployed on a single cluster, you can configure queryable backups. Queryable backups aren't supported for multi-cluster Ops Manager deployments.