MongoDB Ops Manager 데이터 백업, 데이터베이스 성능 모니터링 등과 같은 워크로드를 원활하게 처리하는 역할을 합니다. 멀티 클러스터 MongoDB Ops Manager 및 애플리케이션 데이터베이스 배포서버 전체 데이터 센터 또는 구역 장애에 대해 복원력 있게 만들려면 MongoDB Ops Manager 애플리케이션 및 애플리케이션 데이터베이스를 여러 Kubernetes 클러스터에 배포 .
전제 조건
다음 절차를 시작하기 전에 다음 조치를 수행하세요.
kubectl설치.GKE 클러스터 절차 또는 이에 상응하는 절차를 완료합니다.
TLS 인증서 절차 또는 이에 상응하는 절차를 완료합니다.
ExternalDNS 절차 또는 이에 상응하는 절차를 완료합니다.
MongoDB 연산자 배포 절차를 완료합니다.
다음과 같이 필수 환경 변수를 설정합니다.
# This script builds on top of the environment configured in the setup guides.# It depends (uses) the following env variables defined there to work correctly.# If you don't use the setup guide to bootstrap the environment, then define them here.# ${K8S_CLUSTER_0_CONTEXT_NAME}# ${K8S_CLUSTER_1_CONTEXT_NAME}# ${K8S_CLUSTER_2_CONTEXT_NAME}# ${OM_NAMESPACE}# ${CUSTOM_DOMAIN}# ${DNS_ZONE}
# Defaults for the RustFS setup module and the MongoDBOpsManager backup stores.# If you use your own S3 storage - override any of these.export S3_OPLOG_BUCKET_NAME="${S3_OPLOG_BUCKET_NAME:-s3-oplog-store}"export S3_SNAPSHOT_BUCKET_NAME="${S3_SNAPSHOT_BUCKET_NAME:-s3-snapshot-store}"export S3_ENDPOINT="${S3_ENDPOINT:-rustfs.rustfs.svc.cluster.local}"export S3_ACCESS_KEY="${S3_ACCESS_KEY:-rustfsadmin}"export S3_SECRET_KEY="${S3_SECRET_KEY:-rustfsadmin123}"
export OPS_MANAGER_VERSION="8.0.5"export APPDB_VERSION="8.0.5-ent"
export OPS_MANAGER_EXTERNAL_DOMAIN="opsmanager.${CUSTOM_DOMAIN}"export APPDB_CLUSTER_0_EXTERNAL_DOMAIN="${K8S_CLUSTER_0}.${CUSTOM_DOMAIN}"export APPDB_CLUSTER_1_EXTERNAL_DOMAIN="${K8S_CLUSTER_1}.${CUSTOM_DOMAIN}"export APPDB_CLUSTER_2_EXTERNAL_DOMAIN="${K8S_CLUSTER_2}.${CUSTOM_DOMAIN}"
# Run-scoped names for project-global GCP load balancer resources (KUBE-268).# Multiple CI runs share the same GCP project; with fixed global names, one run's# pre-clean/teardown deletes a concurrent run's load balancer mid-flight.# Derive the run-specific suffix from ${DNS_ZONE} (set in the ExternalDNS setup# guide). For docs users there is no run suffix and the names stay unchanged.lb_suffix="${DNS_ZONE#"mongodb"}"export OM_FIREWALL_RULE_NAME="fw-ops-manager-hc${lb_suffix}"export OM_HEALTHCHECK_NAME="om-healthcheck${lb_suffix}"export OM_BACKEND_SERVICE_NAME="om-backend-service${lb_suffix}"export OM_URL_MAP_NAME="om-url-map${lb_suffix}"export OM_LB_PROXY_NAME="om-lb-proxy${lb_suffix}"export OM_CERTIFICATE_NAME="om-certificate${lb_suffix}"export OM_FORWARDING_RULE_NAME="om-forwarding-rule${lb_suffix}"
# Retry wrapper for gcloud commands to handle transient GCP API errors# (ConnectionError, RemoteDisconnected, etc.). Functions are inherited by# subshells, so this is available inside snippet functions run by sample_test_runner.gcloud_retry() { for attempt in 1 2 3; do if gcloud "$@"; then return 0; fi if (( attempt < 3 )); then echo "gcloud failed (attempt ${attempt}/3), retrying in $((attempt * 5))s..." >&2 sleep $((attempt * 5)) fi done return 1}소스 코드
포함된 모든 소스 코드 MongoDB Kubernetes Operator 리포지토리 에서 찾을 수 있습니다.
절차
TLS 인증서를 생성합니다.
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" apply -f - <<EOFapiVersion: cert-manager.io/v1kind: Certificatemetadata: name: om-certspec: dnsNames: - ${OPS_MANAGER_EXTERNAL_DOMAIN} duration: 240h0m0s issuerRef: name: my-ca-issuer kind: ClusterIssuer renewBefore: 120h0m0s secretName: cert-prefix-om-cert usages: - server auth - client auth---apiVersion: cert-manager.io/v1kind: Certificatemetadata: name: om-db-certspec: dnsNames: - "*.${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" - "*.${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" - "*.${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" duration: 240h0m0s issuerRef: name: my-ca-issuer kind: ClusterIssuer renewBefore: 120h0m0s secretName: cert-prefix-om-db-cert usages: - server auth - client authEOFGCP 에 TLS 인증서를 추가합니다.
mkdir -p certs
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get secret cert-prefix-om-cert -o jsonpath="{.data['tls\.crt']}" | base64 --decode > certs/tls.crtkubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get secret cert-prefix-om-cert -o jsonpath="{.data['tls\.key']}" | base64 --decode > certs/tls.key
gcloud_retry compute ssl-certificates create "${OM_CERTIFICATE_NAME}" --certificate=certs/tls.crt --private-key=certs/tls.key로드 밸런서 에 필요한 Kubernetes 구성 요소를 만듭니다.
이 로드 밸런서 모든 3 클러스터에 걸쳐 MongoDB Ops Manager 의 모든 복제본 간에 트래픽을 분산합니다.
gcloud_retry compute firewall-rules create "${OM_FIREWALL_RULE_NAME}" \ --action=allow \ --direction=ingress \ --target-tags=mongodb \ --source-ranges=130.211.0.0/22,35.191.0.0/16 \ --rules=tcp:8443
gcloud_retry compute health-checks create https "${OM_HEALTHCHECK_NAME}" \ --use-serving-port \ --request-path=/monitor/health
gcloud_retry compute backend-services create "${OM_BACKEND_SERVICE_NAME}" \ --protocol HTTPS \ --health-checks "${OM_HEALTHCHECK_NAME}" \ --global
gcloud_retry compute url-maps create "${OM_URL_MAP_NAME}" \ --default-service "${OM_BACKEND_SERVICE_NAME}"
gcloud_retry compute target-https-proxies create "${OM_LB_PROXY_NAME}" \ --url-map "${OM_URL_MAP_NAME}" \ --ssl-certificates="${OM_CERTIFICATE_NAME}"
gcloud_retry compute forwarding-rules create "${OM_FORWARDING_RULE_NAME}" \ --global \ --target-https-proxy="${OM_LB_PROXY_NAME}" \ --ports=443NAME NETWORK DIRECTION PRIORITY ALLOW DENY DISABLEDfw-ops-manager-hc-6abb9ad2xf80-15370 default INGRESS 1000 tcp:8443 FalseNAME PROTOCOLom-healthcheck-6abb9ad2xf80-15370 HTTPSNAME BACKENDS PROTOCOLom-backend-service-6abb9ad2xf80-15370 HTTPSNAME DEFAULT_SERVICEom-url-map-6abb9ad2xf80-15370 backendServices/om-backend-service-6abb9ad2xf80-15370NAME SSL_CERTIFICATES URL_MAP REGION CERTIFICATE_MAPom-lb-proxy-6abb9ad2xf80-15370 om-certificate-6abb9ad2xf80-15370 om-url-map-6abb9ad2xf80-15370MongoDB Ops Manager 관리자의 자격 증명 생성합니다.
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" --namespace "${OM_NAMESPACE}" create secret generic om-admin-user-credentials \ --from-literal=Username="admin" \ --from-literal=Password="Passw0rd@" \ --from-literal=FirstName="Jane" \ --from-literal=LastName="Doe"MongoDB Ops Manager 배포합니다.
kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOFapiVersion: mongodb.com/v1kind: MongoDBOpsManagermetadata: name: omspec: topology: MultiCluster version: "${OPS_MANAGER_VERSION}" adminCredentials: om-admin-user-credentials externalConnectivity: type: ClusterIP annotations: cloud.google.com/neg: '{"exposed_ports": {"8443":{}}}' opsManagerURL: "https://${OPS_MANAGER_EXTERNAL_DOMAIN}" security: certsSecretPrefix: cert-prefix tls: ca: ca-issuer clusterSpecList: - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" members: 1 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" members: 2 applicationDatabase: version: "${APPDB_VERSION}" topology: MultiCluster security: certsSecretPrefix: cert-prefix tls: ca: ca-issuer clusterSpecList: - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" members: 2 externalAccess: externalDomain: "${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" members: 2 externalAccess: externalDomain: "${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}" members: 1 externalAccess: externalDomain: "${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" backup: enabled: falseEOFKubernetes Operator가 보류 중 상태 될 때까지 기다립니다.
애플리케이션 데이터베이스와 MongoDB Ops Manager 배포가 모두 완료될 때까지 기다립니다.
echo "Waiting for Application Database to reach Pending phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s
echo "Waiting for Ops Manager to reach Pending phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Pending opsmanager/om --timeout=600sWaiting for Application Database to reach Pending phase...mongodbopsmanager.mongodb.com/om condition metWaiting for Ops Manager to reach Pending phase...mongodbopsmanager.mongodb.com/om condition met로드 밸런서 서비스를 설정합니다.
svcneg0=$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg -o=jsonpath='{.items[0].metadata.name}')
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" \ wait svcneg "${svcneg0}" --for=condition=Initialized --timeout=300s \ || { kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg "${svcneg0}" -o yaml; exit 1; }
gcloud_retry compute backend-services add-backend "${OM_BACKEND_SERVICE_NAME}" \ --global \ --network-endpoint-group="${svcneg0}" \ --network-endpoint-group-zone="${K8S_CLUSTER_0_ZONE}" \ --balancing-mode RATE --max-rate-per-endpoint 5svcneg1=$(kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg -o=jsonpath='{.items[0].metadata.name}')
kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" \ wait svcneg "${svcneg1}" --for=condition=Initialized --timeout=300s \ || { kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get svcneg "${svcneg1}" -o yaml; exit 1; }
gcloud_retry compute backend-services add-backend "${OM_BACKEND_SERVICE_NAME}" \ --global \ --network-endpoint-group="${svcneg1}" \ --network-endpoint-group-zone="${K8S_CLUSTER_1_ZONE}" \ --balancing-mode RATE --max-rate-per-endpoint 5MongoDB Ops Manager 실행 상태 될 때까지 기다립니다.
echo "Waiting for Application Database to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=1200secho; echo "Waiting for Ops Manager to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=1200secho; echo "MongoDBOpsManager resource"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/omecho; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsecho; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsWaiting for Application Database to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
Waiting for Ops Manager to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
MongoDBOpsManager resourceNAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGSom 8.0.5 Running Running Disabled 18m
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-6abb9ad27d6x076f80-15370NAME READY STATUS RESTARTS AGEom-0-0 1/1 Running 0 15mom-db-0-0 2/2 Running 0 5m14som-db-0-1 2/2 Running 0 6m15s
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-6abb9ad27d6x076f80-15370NAME READY STATUS RESTARTS AGEom-1-0 1/1 Running 0 15mom-1-1 1/1 Running 0 10mom-db-1-0 2/2 Running 0 3m22som-db-1-1 2/2 Running 0 4m20s선택 사항입니다. 테스트를 위해 S3호환 저장소를 배포합니다.
This step provides a script that deploys a simple RustFS instance for testing purposes. You can skip this step if you have AWS S3 or other S3-compatible buckets available.
참고
RustFS는 테스트용으로만 제공됩니다. 그리고 생산 환경에서는 적합하지 않습니다. 생산 환경에서는 자체 S3 저장을 사용하세요.
MongoDB Ops Manager 백업을 위한 Kubernetes 시크릿을 구성합니다.
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-access-secret \ --from-literal=accessKey="${S3_ACCESS_KEY}" \ --from-literal=secretKey="${S3_SECRET_KEY}"
# RustFS serves a cert-manager certificate; OM must trust the CA that signed it.kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" create secret generic s3-ca-cert \ --from-literal=ca.crt="$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${RUSTFS_NAMESPACE}" get secret rustfs-tls -o jsonpath="{.data['ca\.crt']}" | base64 --decode)"MongoDB Ops Manager 에서 S3 백업을 활성화합니다.
kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" -f - <<EOFapiVersion: mongodb.com/v1kind: MongoDBOpsManagermetadata: name: omspec: topology: MultiCluster version: "${OPS_MANAGER_VERSION}" adminCredentials: om-admin-user-credentials externalConnectivity: type: ClusterIP annotations: cloud.google.com/neg: '{"exposed_ports": {"8443":{}}}' opsManagerURL: "https://${OPS_MANAGER_EXTERNAL_DOMAIN}" security: certsSecretPrefix: cert-prefix tls: ca: ca-issuer clusterSpecList: - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" members: 1 backup: members: 0 - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" members: 2 backup: members: 0 - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}" members: 0 backup: members: 1 applicationDatabase: version: "${APPDB_VERSION}" topology: MultiCluster security: certsSecretPrefix: cert-prefix tls: ca: ca-issuer clusterSpecList: - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}" members: 2 externalAccess: externalDomain: "${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_0_EXTERNAL_DOMAIN}" - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}" members: 2 externalAccess: externalDomain: "${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_1_EXTERNAL_DOMAIN}" - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}" members: 1 externalAccess: externalDomain: "${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" externalService: annotations: external-dns.alpha.kubernetes.io/hostname: "{podName}.${APPDB_CLUSTER_2_EXTERNAL_DOMAIN}" backup: enabled: true s3Stores: - name: my-s3-block-store s3SecretRef: name: "s3-access-secret" pathStyleAccessEnabled: true s3BucketEndpoint: "${S3_ENDPOINT}" s3BucketName: "${S3_SNAPSHOT_BUCKET_NAME}" customCertificateSecretRefs: - name: s3-ca-cert key: ca.crt s3OpLogStores: - name: my-s3-oplog-store s3SecretRef: name: "s3-access-secret" s3BucketEndpoint: "${S3_ENDPOINT}" s3BucketName: "${S3_OPLOG_BUCKET_NAME}" pathStyleAccessEnabled: true customCertificateSecretRefs: - name: s3-ca-cert key: ca.crtEOFMongoDB Ops Manager 실행 상태 될 때까지 기다립니다.
echo; echo "Waiting for Backup to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.backup.phase}'=Running opsmanager/om --timeout=1200secho "Waiting for Application Database to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=1200secho; echo "Waiting for Ops Manager to reach Running phase..."kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=1200secho; echo "MongoDBOpsManager resource"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get opsmanager/omecho; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsecho; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsecho; echo "Pods running in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${OM_NAMESPACE}" get podsWaiting for Backup to reach Running phase...mongodbopsmanager.mongodb.com/om condition metWaiting for Application Database to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
Waiting for Ops Manager to reach Running phase...mongodbopsmanager.mongodb.com/om condition met
MongoDBOpsManager resourceNAME REPLICAS VERSION STATE (OPSMANAGER) STATE (APPDB) STATE (BACKUP) AGE WARNINGSom 8.0.5 Running Running Running 21m
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0-6abb9ad27d6x076f80-15370NAME READY STATUS RESTARTS AGEom-0-0 1/1 Running 0 18mom-db-0-0 2/2 Running 0 7m58som-db-0-1 2/2 Running 0 8m59s
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1-6abb9ad27d6x076f80-15370NAME READY STATUS RESTARTS AGEom-1-0 1/1 Running 0 18mom-1-1 1/1 Running 0 12mom-db-1-0 2/2 Running 0 6m5som-db-1-1 2/2 Running 0 7m3s
Pods running in cluster gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2-6abb9ad27d6x076f80-15370NAME READY STATUS RESTARTS AGEom-2-backup-daemon-0 1/1 Running 0 2m35som-db-2-0 2/2 Running 0 5m11sMongoDB 조직을 생성하고 자격 증명 가져옵니다.
자격 증명 구성하려면 MongoDB Ops Manager 조직 생성하고, MongoDB Ops Manager UI 에서 프로그래밍 방식의 API 키를 생성하고, 로드 밸런서 IP 로 시크릿 을 생성해야 합니다. 자세한 학습 은 Kubernetes Operator에 대한 자격 증명 생성 을 참조하세요.