MongoDB Ops Manager负责促进备份数据、监控数据库性能等工作负载。 为了使多集群MongoDB Ops Manager和应用程序数据库部署能够应对整个数据中心或区域故障,请在多个Kubernetes集群上部署MongoDB Ops Manager应用程序和应用程序数据库。
先决条件
在开始以下过程之前,请执行以下操作:
安装
kubectl。完成 GKE 集群程序 或同等程序。
完成 TLS 证书程序 或同等程序。
完成 Istio 服务网格程序 或同等程序。
完成部署MongoDB Operator程序。
按如下方式设置所需的环境变量:
# This script builds on top of the environment configured in the setup guides. # It depends (uses) the following env variables defined there to work correctly. # If you don't use the setup guide to bootstrap the environment, then define them here. # ${K8S_CLUSTER_0_CONTEXT_NAME} # ${K8S_CLUSTER_1_CONTEXT_NAME} # ${K8S_CLUSTER_2_CONTEXT_NAME} # ${OM_NAMESPACE} export S3_OPLOG_BUCKET_NAME=s3-oplog-store export S3_SNAPSHOT_BUCKET_NAME=s3-snapshot-store # If you use your own S3 storage - set the values accordingly. # By default we install Minio to handle S3 storage and here are set the default credentials. export S3_ENDPOINT="minio.tenant-tiny.svc.cluster.local" export S3_ACCESS_KEY="console" export S3_SECRET_KEY="console123" # (Optional) Change the following setting when using the external URL. # This env variable is used in OpenSSL configuration to generate # server certificates for Ops Manager Application. export OPS_MANAGER_EXTERNAL_DOMAIN="om-svc.${OM_NAMESPACE}.svc.cluster.local" export OPS_MANAGER_VERSION="8.0.4" export APPDB_VERSION="8.0.5"
源代码
您可以在MongoDB Kubernetes Operator存储库中找到所有包含的源代码。
步骤
生成 TLS 证书。
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" apply -f - <<EOF apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: om-cert spec: dnsNames: - ${OPS_MANAGER_EXTERNAL_DOMAIN} - om-svc.${OM_NAMESPACE}.svc.cluster.local duration: 240h0m0s issuerRef: name: my-ca-issuer kind: ClusterIssuer renewBefore: 120h0m0s secretName: cert-prefix-om-cert usages: - server auth - client auth --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: om-db-cert spec: dnsNames: - "*.${OM_NAMESPACE}.svc.cluster.local" - "*.om-db-svc.${OM_NAMESPACE}.svc.cluster.local" duration: 240h0m0s issuerRef: name: my-ca-issuer kind: ClusterIssuer renewBefore: 120h0m0s secretName: cert-prefix-om-db-cert usages: - server auth - client auth EOF
安装 Ops Manager
此时,您已准备好用于部署 资源的环境和Kubernetes MongoDB Ops ManagerOperator。
为MongoDB Ops Manager管理员用户创建必要的档案, Kubernetes Operator 在部署MongoDB Ops Manager应用程序实例后将创建该用户:
1 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" --namespace "${OM_NAMESPACE}" create secret generic om-admin-user-credentials \ 2 --from-literal=Username="admin" \ 3 --from-literal=Password="Passw0rd@" \ 4 --from-literal=FirstName="Jane" \ 5 --from-literal=LastName="Doe" 在单个成员集群(也称为操作符集群)上部署尽可能简单的
MongoDBOpsManager自定义资源(启用 TLS)。此部署与单集群模式的部署几乎相同,但将
spec.topology和spec.applicationDatabase.topology设立为MultiCluster。这种部署方式表明,单个 Kubernetes 集群部署是单个 Kubernetes 成员集群上的多 Kubernetes 集群部署的特例。 MongoDB Ops ManagerKubernetes您可以从一开始就在所需数量的 集群上部署 应用程序和应用程序数据库,而不必从只有单个成员Kubernetes 集群的部署开始。
此时,您已准备好MongoDB Ops Manager部署以跨越多个Kubernetes集群,本过程稍后将执行此操作。
1 kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOF 2 apiVersion: mongodb.com/v1 3 kind: MongoDBOpsManager 4 metadata: 5 name: om 6 spec: 7 topology: MultiCluster 8 version: "${OPS_MANAGER_VERSION}" 9 adminCredentials: om-admin-user-credentials 10 externalConnectivity: 11 type: LoadBalancer 12 security: 13 certsSecretPrefix: cert-prefix 14 tls: 15 ca: ca-issuer 16 clusterSpecList: 17 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 18 members: 1 19 applicationDatabase: 20 version: "${APPDB_VERSION}" 21 topology: MultiCluster 22 security: 23 certsSecretPrefix: cert-prefix 24 tls: 25 ca: ca-issuer 26 clusterSpecList: 27 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 28 members: 3 29 backup: 30 enabled: false 31 EOF 等待 Kubernetes Operator 接手工作并进入
status.applicationDatabase.phase=Pending状态。 等待应用程序数据库和MongoDB Ops Manager部署完成。1 echo "Waiting for Application Database to reach Pending phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s 1 Waiting for Application Database to reach Pending phase... 2 mongodbopsmanager.mongodb.com/om condition met 部署MongoDB Ops Manager 。 Kubernetes Operator 通过执行以下步骤来部署MongoDB Ops Manager 。 它:
部署应用程序数据库的副本集节点,并等待副本集中的 MongoDB 进程开始运行。
使用应用程序数据库的连接 部署MongoDB Ops Managerstring 应用程序实例,并等待其准备就绪。
将监控MongoDB Agent容器添加到每个应用程序数据库的 Pod。
等待MongoDB Ops Manager应用程序和应用程序数据库 Pod 开始运行。
1 echo "Waiting for Application Database to reach Running phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s 3 echo; echo "Waiting for Ops Manager to reach Running phase..." 4 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s 5 echo; echo "Waiting for Application Database to reach Pending phase (enabling monitoring)..." 6 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s 7 echo "Waiting for Application Database to reach Running phase..." 8 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s 9 echo; echo "Waiting for Ops Manager to reach Running phase..." 10 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s 11 echo; echo "MongoDBOpsManager resource" 12 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/om 13 echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}" 14 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 15 echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}" 16 kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 1 Waiting for Application Database to reach Running phase... 2 mongodbopsmanager.mongodb.com/om condition met 3 4 Waiting for Ops Manager to reach Running phase... 5 mongodbopsmanager.mongodb.com/om condition met 6 7 Waiting for Application Database to reach Pending phase (enabling monitoring)... 8 mongodbopsmanager.mongodb.com/om condition met 9 Waiting for Application Database to reach Running phase... 10 mongodbopsmanager.mongodb.com/om condition met 11 12 Waiting for Ops Manager to reach Running phase... 13 mongodbopsmanager.mongodb.com/om condition met 14 15 MongoDBOpsManager resource 16 NAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGS 17 om 8.0.4 Running Running Disabled 11m 18 19 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-67d0389d75b70a0007e5894a 20 NAME READY STATUS RESTARTS AGE 21 om-0-0 2/2 Running 0 7m35s 22 om-db-0-0 4/4 Running 0 37s 23 om-db-0-1 4/4 Running 0 115s 24 om-db-0-2 4/4 Running 0 2m57s 25 26 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-67d0389d75b70a0007e5894a 现在您已经以多集群模式部署了一个单成员集群,您可以重新配置此部署以跨越多个 Kubernetes 集群。
在第二个成员集群上,再部署两个应用程序数据库副本集成员和一个MongoDB Ops Manager应用程序实例:
1 kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOF 2 apiVersion: mongodb.com/v1 3 kind: MongoDBOpsManager 4 metadata: 5 name: om 6 spec: 7 topology: MultiCluster 8 version: "${OPS_MANAGER_VERSION}" 9 adminCredentials: om-admin-user-credentials 10 externalConnectivity: 11 type: LoadBalancer 12 security: 13 certsSecretPrefix: cert-prefix 14 tls: 15 ca: ca-issuer 16 clusterSpecList: 17 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 18 members: 1 19 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" 20 members: 1 21 applicationDatabase: 22 version: "${APPDB_VERSION}" 23 topology: MultiCluster 24 security: 25 certsSecretPrefix: cert-prefix 26 tls: 27 ca: ca-issuer 28 clusterSpecList: 29 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 30 members: 3 31 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" 32 members: 2 33 backup: 34 enabled: false 35 EOF 等待 Kubernetes Operator 接手工作(待处理阶段):
1 echo "Waiting for Application Database to reach Pending phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s 1 Waiting for Application Database to reach Pending phase... 2 mongodbopsmanager.mongodb.com/om condition met 等待 Kubernetes Operator 完成所有组件的部署:
1 echo "Waiting for Application Database to reach Running phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s 3 echo; echo "Waiting for Ops Manager to reach Running phase..." 4 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s 5 echo; echo "MongoDBOpsManager resource" 6 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/om 7 echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}" 8 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 9 echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}" 10 kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 1 Waiting for Application Database to reach Running phase... 2 mongodbopsmanager.mongodb.com/om condition met 3 4 Waiting for Ops Manager to reach Running phase... 5 mongodbopsmanager.mongodb.com/om condition met 6 7 MongoDBOpsManager resource 8 NAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGS 9 om 8.0.4 Running Running Disabled 19m 10 11 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-67d0389d75b70a0007e5894a 12 NAME READY STATUS RESTARTS AGE 13 om-0-0 2/2 Running 0 2m35s 14 om-db-0-0 4/4 Running 0 8m35s 15 om-db-0-1 4/4 Running 0 9m53s 16 om-db-0-2 4/4 Running 0 10m 17 18 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-67d0389d75b70a0007e5894a 19 NAME READY STATUS RESTARTS AGE 20 om-1-0 2/2 Running 0 3m5s 21 om-db-1-0 4/4 Running 0 7m49s 22 om-db-1-1 4/4 Running 0 5m54s
启用备份。
在 MongoDB Ops Manager 应用程序的多 Kubernetes 集群部署中,您只能配置基于 S3 的备份存储。此过程引用 env_variables.sh 中定义的 S3_*。
可选。安装 MinIO Operator。
此过程使用 MinIO Operator 为备份部署 S3 兼容存储。如果您有可用的Amazon Web Services S3 或其他 S3 兼容存储桶,则可以跳过此步骤。在这种情况下,请在 env_variables.sh 中相应地调整
S3_*变量。1 kubectl kustomize "github.com/minio/operator/resources/?timeout=120&ref=v5.0.12" | \ 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f - 3 4 kubectl kustomize "github.com/minio/operator/examples/kustomization/tenant-tiny?timeout=120&ref=v5.0.12" | \ 5 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f - 6 7 # add two buckets to the tenant config 8 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "tenant-tiny" patch tenant/myminio \ 9 --type='json' \ 10 -p="[{\"op\": \"add\", \"path\": \"/spec/buckets\", \"value\": [{\"name\": \"${S3_OPLOG_BUCKET_NAME}\"}, {\"name\": \"${S3_SNAPSHOT_BUCKET_NAME}\"}]}]" 11 12 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "tenant-tiny" istio-injection=enabled --overwrite 在配置和启用备份之前,请创建密钥:
s3-access-secret— 包含S 3档案。s3-ca-cert- 包含颁发存储桶服务器证书的CA证书。 在此过程中使用的示例 MinIO 部署中,将使用默认的 Kubernetes 根CA证书对证书进行签名。 由于它不是公众信任的CA证书,因此您必须提供该证书, MongoDB Ops Manager才能信任该连接。
如果您使用公开信任的证书,则可以跳过此步骤并删除
spec.backup.s3Stores.customCertificateSecretRefs和spec.backup.s3OpLogStores.customCertificateSecretRefs设置中的值。1 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-access-secret \ 2 --from-literal=accessKey="${S3_ACCESS_KEY}" \ 3 --from-literal=secretKey="${S3_SECRET_KEY}" 4 5 # minio TLS secrets are signed with the default k8s root CA 6 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-ca-cert \ 7 --from-literal=ca.crt="$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n kube-system get configmap kube-root-ca.crt -o jsonpath="{.data.ca\.crt}")"
在启用备份的情况下重新部署MongoDB Ops Manager 。
KubernetesOperatorMongoDB Ops Manager 可以在为其配置Kubernetes Operator 的任何成员集群上以任意组合配置和部署所有组件、 应用程序、备份守护程序实例和应用程序数据库的副本集节点。
为了说明多 Kubernetes 集群部署配置的灵活性,在第三个成员集群上仅部署一个备份守护进程实例,并为第一个和第二个集群指定零个备份守护进程成员。
1 kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOF 2 apiVersion: mongodb.com/v1 3 kind: MongoDBOpsManager 4 metadata: 5 name: om 6 spec: 7 topology: MultiCluster 8 version: "${OPS_MANAGER_VERSION}" 9 adminCredentials: om-admin-user-credentials 10 externalConnectivity: 11 type: LoadBalancer 12 security: 13 certsSecretPrefix: cert-prefix 14 tls: 15 ca: ca-issuer 16 clusterSpecList: 17 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 18 members: 1 19 backup: 20 members: 0 21 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" 22 members: 1 23 backup: 24 members: 0 25 - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}" 26 members: 0 27 backup: 28 members: 1 29 configuration: # to avoid configuration wizard on first login 30 mms.adminEmailAddr: email@example.com 31 mms.fromEmailAddr: email@example.com 32 mms.ignoreInitialUiSetup: "true" 33 mms.mail.hostname: smtp@example.com 34 mms.mail.port: "465" 35 mms.mail.ssl: "true" 36 mms.mail.transport: smtp 37 mms.minimumTLSVersion: TLSv1.2 38 mms.replyToEmailAddr: email@example.com 39 applicationDatabase: 40 version: "${APPDB_VERSION}" 41 topology: MultiCluster 42 security: 43 certsSecretPrefix: cert-prefix 44 tls: 45 ca: ca-issuer 46 clusterSpecList: 47 - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" 48 members: 3 49 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" 50 members: 2 51 backup: 52 enabled: true 53 s3Stores: 54 - name: my-s3-block-store 55 s3SecretRef: 56 name: "s3-access-secret" 57 pathStyleAccessEnabled: true 58 s3BucketEndpoint: "${S3_ENDPOINT}" 59 s3BucketName: "${S3_SNAPSHOT_BUCKET_NAME}" 60 customCertificateSecretRefs: 61 - name: s3-ca-cert 62 key: ca.crt 63 s3OpLogStores: 64 - name: my-s3-oplog-store 65 s3SecretRef: 66 name: "s3-access-secret" 67 s3BucketEndpoint: "${S3_ENDPOINT}" 68 s3BucketName: "${S3_OPLOG_BUCKET_NAME}" 69 pathStyleAccessEnabled: true 70 customCertificateSecretRefs: 71 - name: s3-ca-cert 72 key: ca.crt 73 EOF 等待 Kubernetes Operator 完成配置:
1 echo; echo "Waiting for Backup to reach Running phase..." 2 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.backup.phase}'=Running opsmanager/om --timeout=1200s 3 echo "Waiting for Application Database to reach Running phase..." 4 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s 5 echo; echo "Waiting for Ops Manager to reach Running phase..." 6 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s 7 echo; echo "MongoDBOpsManager resource" 8 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/om 9 echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}" 10 kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 11 echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}" 12 kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 13 echo; echo "Pods running in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}" 14 kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get pods 1 Waiting for Backup to reach Running phase... 2 mongodbopsmanager.mongodb.com/om condition met 3 Waiting for Application Database to reach Running phase... 4 mongodbopsmanager.mongodb.com/om condition met 5 6 Waiting for Ops Manager to reach Running phase... 7 mongodbopsmanager.mongodb.com/om condition met 8 9 MongoDBOpsManager resource 10 NAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGS 11 om 8.0.4 Running Running Running 24m 12 13 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-67d0389d75b70a0007e5894a 14 NAME READY STATUS RESTARTS AGE 15 om-0-0 2/2 Running 0 3m56s 16 om-db-0-0 4/4 Running 0 13m 17 om-db-0-1 4/4 Running 0 14m 18 om-db-0-2 4/4 Running 0 15m 19 20 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-67d0389d75b70a0007e5894a 21 NAME READY STATUS RESTARTS AGE 22 om-1-0 2/2 Running 0 3m55s 23 om-db-1-0 4/4 Running 0 12m 24 om-db-1-1 4/4 Running 0 10m 25 26 Pods running in cluster gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2-67d0389d75b70a0007e5894a 27 NAME READY STATUS RESTARTS AGE 28 om-2-backup-daemon-0 2/2 Running 0 113s
为 Kubernetes 操作符 创建档案。
要配置凭证,您必须创建一个MongoDB Ops Manager组织,在MongoDB Ops Manager用户界面中生成编程API密钥,并使用您的负载均衡器IP创建密钥。请参阅为Kubernetes Operator 创建档案以学习;了解更多信息。