Deploy a Multi-Cluster External Application Database

You can deploy an Ops Manager resource that uses a multi-cluster external Application Database: a MongoDBMultiCluster custom resource with spec.role set to AppDB. This resource spreads the Application Database members across multiple Kubernetes clusters. A second Ops Manager instance manages the resource as a project, which lets Ops Manager back up and restore the Application Database. To learn more, see Back Up the Ops Manager Application Database.

This procedure deploys a new Ops Manager instance. To convert an existing Ops Manager instance that uses an internally managed multi-cluster Application Database, see Migrate a Multi-Cluster Application Database. For the single-cluster variant of this procedure, see Deploy Ops Manager with an External Application Database.

This procedure deploys the following resources:

A management Ops Manager instance with its own internally managed Application Database, running on the central cluster. This instance manages the project that contains the external Application Database.
An external Application Database: a MongoDBMultiCluster replica set with spec.role set to AppDB, named <primary-om-name>-db. The Kubernetes Operator creates one StatefulSet per member cluster, named <primary-om-name>-db-<cluster-index>.
A primary Ops Manager instance that omits spec.applicationDatabase and references the external Application Database with spec.externalApplicationDatabaseRef set to kind MongoDBMultiCluster.

Prerequisites

Before you begin, complete the following tasks:

Deploy three Kubernetes clusters, each with a default StorageClass: one central cluster and two member clusters.
Deploy a service mesh that spans the clusters and provides cross-cluster service discovery. The Application Database members communicate across clusters over the mesh.

This procedure uses Istio, but you can use any service mesh that provides FQDN hostname resolution between Pods across clusters. A service mesh isn't a requirement for a multi-cluster external Application Database itself. For a no-mesh deployment pattern, see Multi-Cluster Ops Manager Without a Service Mesh.
Install kubectl, helm, and the kubectl mongodb plugin, and configure them for all three clusters.
Obtain access to the Kubernetes Operator Helm chart.

Considerations

Review the following considerations before you deploy:

Set the Ops Manager version and the Application Database version to a consistent pair. The management Ops Manager instance manages the external Application Database, so the Application Database version must be a MongoDB version that the management Ops Manager instance offers in its version manifest. For example, an 8.0.x Ops Manager instance offers 8.0.x MongoDB versions, but a 7.0.x Ops Manager instance doesn't. To find a compatible pair, see the Application Database compatibility matrix.
A multi-cluster external Application Database requires TLS encryption. When the Application Database uses TLS, the MongoDB Agent validates every outbound HTTPS connection against the CA bundle in spec.security.tls.ca, including the download of the MongoDB binary from downloads.mongodb.com. The CA bundle must include the public roots that sign downloads.mongodb.com in addition to your own CA. To configure TLS certificates for a multi-cluster deployment, see Manage TLS Certificates.
The central cluster runs the Kubernetes Operator, both Ops Manager instances, and holds the MongoDBMultiCluster resource. The member clusters run the Application Database data pods.

Procedure

Set the environment variables for the deployment.

export K8S_CTX_0="<central-cluster-context>"export K8S_CTX_1="<member-cluster-1-context>"export K8S_CTX_2="<member-cluster-2-context>"export MDB_NS="mongodb"
export OPERATOR_HELM_CHART="oci://quay.io/mongodb/helm-charts/mongodb-kubernetes"
export OM_VERSION="8.0.7"export APPDB_VERSION="8.0.5-ent"
export MANAGEMENT_OM_NAME="management-om"export PRIMARY_OM_NAME="primary-om"
export APPDB_NAME="${PRIMARY_OM_NAME}-db"export APPDB_MEMBERS_C1=2export APPDB_MEMBERS_C2=2
export MANAGEMENT_OM_URL="http://${MANAGEMENT_OM_NAME}-svc.${MDB_NS}.svc.cluster.local:8080"export MANAGEMENT_OM_ADMIN_KEY_SECRET="${MDB_NS}-${MANAGEMENT_OM_NAME}-admin-key"
export APPDB_PROJECT_CONFIGMAP="${APPDB_NAME}-config"export APPDB_PROJECT_NAME="external-appdb"export APPDB_CONNECTION_STRING_SECRET="${APPDB_NAME}-connection-string"
export CERT_MANAGER_NAMESPACE="cert-manager"export APPDB_TLS_SELF_SIGNED_ISSUER="mongodb-self-signed-issuer"export APPDB_TLS_CA_CERT_NAME="mongodb-ca"export APPDB_TLS_CA_SECRET_NAME="mongodb-ca-key-pair"export APPDB_TLS_CA_ISSUER="mongodb-ca-issuer"export APPDB_CERT_PREFIX="appdb"export APPDB_CA_CONFIGMAP="${APPDB_NAME}-ca"
export OM_ADMIN_EMAIL="admin@example.com"export OM_ADMIN_PASSWORD="<your-password>"export OM_ADMIN_FIRST_NAME="Admin"export OM_ADMIN_LAST_NAME="User"

The Kubernetes Operator requires the external Application Database to be named <primary-om-name>-db, which is why APPDB_NAME derives from PRIMARY_OM_NAME.

Create the namespace in each cluster.

The namespace must exist on the central cluster and on both member clusters. The istio-injection=enabled label adds a sidecar to the Application Database pods so that their agents can resolve the management Ops Manager service across clusters.

for ctx in "${K8S_CTX_0}" "${K8S_CTX_1}" "${K8S_CTX_2}"; do  kubectl create namespace "${MDB_NS}" --context "${ctx}" --dry-run=client -o yaml \    | kubectl apply --context "${ctx}" -f -  kubectl label namespace "${MDB_NS}" istio-injection=enabled --overwrite --context "${ctx}"done

Install the Kubernetes Operator in multi-cluster mode.

Configure service accounts and roles across the clusters with the kubectl mongodb plugin, then install the Kubernetes Operator on the central cluster with the member clusters registered. To learn more, see Multi-Cluster Deployment Prerequisites.

kubectl mongodb multicluster setup \  --central-cluster="${K8S_CTX_0}" \  --member-clusters="${K8S_CTX_0},${K8S_CTX_1},${K8S_CTX_2}" \  --member-cluster-namespace="${MDB_NS}" \  --central-cluster-namespace="${MDB_NS}" \  --create-service-account-secrets \  --install-database-roles=true
helm upgrade --install --kube-context "${K8S_CTX_0}" \  --create-namespace --namespace "${MDB_NS}" \  mongodb-kubernetes-operator-multi-cluster \  --set operator.name=mongodb-kubernetes-operator-multi-cluster \  --set operator.createOperatorServiceAccount=false \  --set operator.createResourcesServiceAccountsAndRoles=false \  --set "multiCluster.clusters={${K8S_CTX_0},${K8S_CTX_1},${K8S_CTX_2}}" \  "${OPERATOR_HELM_CHART}"
kubectl rollout status deployment/mongodb-kubernetes-operator-multi-cluster \  --context "${K8S_CTX_0}" -n "${MDB_NS}" --timeout=300s

Create the Ops Manager admin secret.

Both Ops Manager resources reference this secret with spec.adminCredentials.

kubectl create secret generic ops-manager-admin-secret \  --context "${K8S_CTX_0}" -n "${MDB_NS}" \  --from-literal=Username="${OM_ADMIN_EMAIL}" \  --from-literal=Password="${OM_ADMIN_PASSWORD}" \  --from-literal=FirstName="${OM_ADMIN_FIRST_NAME}" \  --from-literal=LastName="${OM_ADMIN_LAST_NAME}"

Install cert-manager on the central cluster.

The multi-cluster Application Database members use TLS, so cert-manager must issue the CA and the Application Database server certificate.

helm upgrade --install --kube-context "${K8S_CTX_0}" \  cert-manager jetstack/cert-manager \  --namespace "${CERT_MANAGER_NAMESPACE}" --create-namespace \  --set crds.enabled=true

Configure the TLS CA.

Create the signing chain, then publish the CA into a ConfigMap that the Kubernetes Operator references as spec.security.tls.ca. The bundle combines your self-signed CA with the host's public CA bundle, because the MongoDB Agent validates the MongoDB binary download from downloads.mongodb.com against this bundle. The combined bundle exceeds the size limit of the client-side kubectl apply annotation, so apply the ConfigMap server-side.

kubectl apply --context "${K8S_CTX_0}" -f - <<EOFapiVersion: cert-manager.io/v1kind: ClusterIssuermetadata:  name: ${APPDB_TLS_SELF_SIGNED_ISSUER}spec:  selfSigned: {}EOF
kubectl apply --context "${K8S_CTX_0}" -n "${CERT_MANAGER_NAMESPACE}" -f - <<EOFapiVersion: cert-manager.io/v1kind: Certificatemetadata:  name: ${APPDB_TLS_CA_CERT_NAME}spec:  isCA: true  commonName: mongodb-ca  secretName: ${APPDB_TLS_CA_SECRET_NAME}  duration: 87600h  renewBefore: 8760h  privateKey:    algorithm: ECDSA    size: 256  issuerRef:    name: ${APPDB_TLS_SELF_SIGNED_ISSUER}    kind: ClusterIssuerEOF
kubectl wait --for=condition=Ready certificate/"${APPDB_TLS_CA_CERT_NAME}" \  -n "${CERT_MANAGER_NAMESPACE}" --context "${K8S_CTX_0}" --timeout=120s
kubectl apply --context "${K8S_CTX_0}" -f - <<EOFapiVersion: cert-manager.io/v1kind: ClusterIssuermetadata:  name: ${APPDB_TLS_CA_ISSUER}spec:  ca:    secretName: ${APPDB_TLS_CA_SECRET_NAME}EOF
ca_crt=$(kubectl get secret "${APPDB_TLS_CA_SECRET_NAME}" \  -n "${CERT_MANAGER_NAMESPACE}" --context "${K8S_CTX_0}" \  -o jsonpath='{.data.ca\.crt}' | base64 -d)
system_ca=""for candidate in /etc/ssl/certs/ca-certificates.crt /etc/pki/tls/certs/ca-bundle.crt /etc/ssl/cert.pem; do  [[ -r "${candidate}" ]] && { system_ca=$(cat "${candidate}"); break; }done
ca_bundle_file=$(mktemp)printf '%s\n%s\n' "${ca_crt}" "${system_ca}" > "${ca_bundle_file}"
kubectl create configmap "${APPDB_CA_CONFIGMAP}" \  --context "${K8S_CTX_0}" -n "${MDB_NS}" \  --from-file=ca-pem="${ca_bundle_file}" \  --from-file=mms-ca.crt="${ca_bundle_file}" \  --dry-run=client -o yaml \  | kubectl apply --server-side --force-conflicts --context "${K8S_CTX_0}" -n "${MDB_NS}" -f -
rm -f "${ca_bundle_file}"

Generate the Application Database server certificate.

The certificate secret name follows the Kubernetes Operator convention <certs-secret-prefix>-<appdb-name>-cert. The wildcard dnsNames entry covers the service FQDN of every Application Database member in every cluster.

kubectl apply --context "${K8S_CTX_0}" -n "${MDB_NS}" -f - <<EOFapiVersion: cert-manager.io/v1kind: Certificatemetadata:  name: ${APPDB_CERT_PREFIX}-${APPDB_NAME}-certspec:  secretName: ${APPDB_CERT_PREFIX}-${APPDB_NAME}-cert  duration: 8760h  renewBefore: 720h  privateKey:    algorithm: RSA    size: 2048  usages: ["server auth", "client auth"]  dnsNames:    - "*.${MDB_NS}.svc.cluster.local"  issuerRef:    name: ${APPDB_TLS_CA_ISSUER}    kind: ClusterIssuerEOF
kubectl wait --for=condition=Ready certificate/"${APPDB_CERT_PREFIX}-${APPDB_NAME}-cert" \  -n "${MDB_NS}" --context "${K8S_CTX_0}" --timeout=120s

Deploy the management Ops Manager instance.

kubectl apply --context "${K8S_CTX_0}" -n "${MDB_NS}" -f - <<EOFapiVersion: mongodb.com/v1kind: MongoDBOpsManagermetadata:  name: ${MANAGEMENT_OM_NAME}spec:  replicas: 1  version: ${OM_VERSION}  adminCredentials: ops-manager-admin-secret  applicationDatabase:    members: 3    version: ${APPDB_VERSION}  backup:    enabled: false  configuration:    automation.versions.source: mongodb    mms.ignoreInitialUiSetup: "true"    mms.adminEmailAddr: admin@example.com    mms.fromEmailAddr: admin@example.com    mms.replyToEmailAddr: admin@example.com    mms.mail.hostname: email-smtp.us-east-1.amazonaws.com    mms.mail.port: "465"    mms.mail.ssl: "true"    mms.mail.transport: smtp    mms.minimumTLSVersion: TLSv1.2EOF

The mms.* mail settings are required. When mms.ignoreInitialUiSetup is true, the Ops Manager pre-flight check doesn't start Ops Manager unless mms.fromEmailAddr and the related mail settings are present.

Wait for the management Ops Manager instance to become ready.

Wait for the internally managed Application Database and the Ops Manager resource:
```
kubectl wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running \  om/"${MANAGEMENT_OM_NAME}" --context "${K8S_CTX_0}" -n "${MDB_NS}" --timeout=1200s
kubectl wait --for=jsonpath='{.status.opsManager.phase}'=Running \  om/"${MANAGEMENT_OM_NAME}" --context "${K8S_CTX_0}" -n "${MDB_NS}" --timeout=1800s
```
Confirm that the Ops Manager public API answers requests. An opsManager.phase of Running means that the Ops Manager pod is up, but the API that the Kubernetes Operator uses to create the Application Database project can lag by a few seconds. A 401 response is the expected unauthenticated response and means that the API is ready:
```
om_pod="${MANAGEMENT_OM_NAME}-0"until [ "$(kubectl exec "${om_pod}" -c mongodb-ops-manager --context "${K8S_CTX_0}" -n "${MDB_NS}" -- \  curl -s -o /dev/null -w '%{http_code}' \  "http://$(kubectl get pod "${om_pod}" --context "${K8S_CTX_0}" -n "${MDB_NS}" \    -o jsonpath='{.status.podIP}'):8080/api/public/v1.0" 2>/dev/null)" = "401" ]; do  echo "waiting for management Ops Manager public API..."; sleep 10done
```
Confirm that the Kubernetes Operator created the programmatic API key secret for the management Ops Manager instance:
```
kubectl get secret "${MANAGEMENT_OM_ADMIN_KEY_SECRET}" \  --context "${K8S_CTX_0}" -n "${MDB_NS}"
```

Create the project ConfigMap for the external Application Database.

The sslMMSCAConfigMap entry advertises the CA that the TLS-enabled Application Database agents must trust when they connect to the management Ops Manager instance.

kubectl create configmap "${APPDB_PROJECT_CONFIGMAP}" \  --context "${K8S_CTX_0}" -n "${MDB_NS}" \  --from-literal=baseUrl="${MANAGEMENT_OM_URL}" \  --from-literal=projectName="${APPDB_PROJECT_NAME}" \  --from-literal=sslMMSCAConfigMap="${APPDB_CA_CONFIGMAP}" \  --from-literal=orgId="" \  --dry-run=client -o yaml \  | kubectl apply --context "${K8S_CTX_0}" -n "${MDB_NS}" -f -

Create the external Application Database.

Apply the MongoDBMultiCluster resource on the central cluster. The Kubernetes Operator creates the Application Database members on the member clusters.

kubectl apply --context "${K8S_CTX_0}" -n "${MDB_NS}" -f - <<EOFapiVersion: mongodb.com/v1kind: MongoDBMultiClustermetadata:  name: ${APPDB_NAME}spec:  version: ${APPDB_VERSION}  type: ReplicaSet  role: AppDB  persistent: true  credentials: ${MANAGEMENT_OM_ADMIN_KEY_SECRET}  opsManager:    configMapRef:      name: ${APPDB_PROJECT_CONFIGMAP}  security:    certsSecretPrefix: ${APPDB_CERT_PREFIX}    tls:      enabled: true      ca: ${APPDB_CA_CONFIGMAP}  clusterSpecList:    - clusterName: ${K8S_CTX_1}      members: ${APPDB_MEMBERS_C1}    - clusterName: ${K8S_CTX_2}      members: ${APPDB_MEMBERS_C2}EOF
kubectl wait --for=jsonpath='{.status.phase}'=Running \  mdbmc/"${APPDB_NAME}" --context "${K8S_CTX_0}" -n "${MDB_NS}" --timeout=1200s

When you set spec.role to AppDB, the Kubernetes Operator configures SCRAM authentication for the resource. To learn more, see spec.role.

Deploy the primary Ops Manager instance.

This resource omits spec.applicationDatabase and references the external Application Database with kind set to MongoDBMultiCluster.

kubectl apply --context "${K8S_CTX_0}" -n "${MDB_NS}" -f - <<EOFapiVersion: mongodb.com/v1kind: MongoDBOpsManagermetadata:  name: ${PRIMARY_OM_NAME}spec:  replicas: 1  version: ${OM_VERSION}  adminCredentials: ops-manager-admin-secret  externalApplicationDatabaseRef:    name: ${APPDB_NAME}    kind: MongoDBMultiCluster  backup:    enabled: false  configuration:    automation.versions.source: mongodb    mms.ignoreInitialUiSetup: "true"    mms.adminEmailAddr: admin@example.com    mms.fromEmailAddr: admin@example.com    mms.replyToEmailAddr: admin@example.com    mms.mail.hostname: email-smtp.us-east-1.amazonaws.com    mms.mail.port: "465"    mms.mail.ssl: "true"    mms.mail.transport: smtp    mms.minimumTLSVersion: TLSv1.2EOF

To enable backup for your MongoDB deployments, set spec.backup.enabled to true on this resource and configure snapshot storage. To learn more, see Configure File System Backup Store with Kubernetes Operator.

Wait for the primary Ops Manager instance to become ready.

kubectl wait --for=jsonpath='{.status.opsManager.phase}'=Running \  om/"${PRIMARY_OM_NAME}" --context "${K8S_CTX_0}" -n "${MDB_NS}" --timeout=1800s
kubectl wait --for=jsonpath='{.status.applicationDatabase.phase}'=Disabled \  om/"${PRIMARY_OM_NAME}" --context "${K8S_CTX_0}" -n "${MDB_NS}" --timeout=600s

The Kubernetes Operator doesn't manage an internal Application Database for an Ops Manager resource that uses an external Application Database, so status.applicationDatabase.phase reports Disabled. This is expected and isn't an error.

Verify the deployment.

In a multi-cluster deployment, the Application Database StatefulSets run in the member clusters, and the Kubernetes Operator tracks their ownership with the mongodbmulticluster label instead of an owner reference.

Confirm that the Application Database pods are running in each member cluster and that each StatefulSet carries the mongodbmulticluster label set to <namespace>-<appdb-name>:
```
idx=0for ctx in "${K8S_CTX_1}" "${K8S_CTX_2}"; do  sts="${APPDB_NAME}-${idx}"  kubectl get pods --context "${ctx}" -n "${MDB_NS}" \    -l "mongodbmulticluster=${MDB_NS}-${APPDB_NAME}"  kubectl get statefulset "${sts}" --context "${ctx}" -n "${MDB_NS}" \    -o jsonpath='{.metadata.labels.mongodbmulticluster}{"\n"}'  idx=$((idx + 1))done
```
Confirm that the Kubernetes Operator created the connection string secret for the primary Ops Manager instance:
```
kubectl get secret "${APPDB_CONNECTION_STRING_SECRET}" \  --context "${K8S_CTX_0}" -n "${MDB_NS}"
```

Common External Application Database Issues

The Application Database Agents Fail the Binary Download

If the Application Database agents fail to download the MongoDB binary with x509: certificate signed by unknown authority, the CA bundle in spec.security.tls.ca doesn't include the public roots that sign downloads.mongodb.com. With TLS enabled, the MongoDB Agent validates the download against this bundle instead of the system trust store. Recreate the ConfigMap with both your self-signed CA and the host's public CA bundle, as the TLS CA step shows.

The Application Database Pods Can't Resolve Ops Manager

If the Application Database agents log Could not resolve host for the management Ops Manager service, the member-cluster namespace is missing the istio-injection=enabled label. Without the sidecar, the pods can't resolve the central-cluster Ops Manager service over the mesh. Apply the label as the namespace step shows.

The CA ConfigMap Is Rejected as Too Long

If Kubernetes rejects the CA ConfigMap with metadata.annotations: Too long, the combined CA bundle exceeds the size limit of the client-side kubectl apply annotation. Apply the ConfigMap with kubectl apply --server-side --force-conflicts, or create it with kubectl create.

The MongoDBMultiCluster Resource Reports a Version Error

If the external Application Database reports Failed with a message that the MongoDB version isn't available, set spec.version to a version that the management Ops Manager instance offers. Keep the Ops Manager version and the Application Database version on the same major version.

The Operator Rejects the External Application Database Reference

The value of spec.externalApplicationDatabaseRef.name must be exactly <primary-om-name>-db, and the MongoDBMultiCluster resource must be in the same namespace as the Ops Manager resource.

Next Steps

To back up the external Application Database, enable backup on the MongoDBMultiCluster resource in the project that the management Ops Manager instance manages. To learn more, see Configure MongoDB Database Backups.

Quick Start