여러 Kubernetes 클러스터에 MongoDB Ops Manager 리소스 배포

Kubernetes 클러스터를 생성합니다.

서비스 메시를 사용하여 자체 Kubernetes 클러스터를 이미 설치 및 구성한 경우 이 단계를 건너뛸 수 있습니다.

1. 3개의 GKE(Google Kubernetes Engine) 클러스터를 만듭니다.

   1
   gcloud container clusters create "${K8S_CLUSTER_0}" \
   2
         --zone="${K8S_CLUSTER_0_ZONE}" \
   3
         --num-nodes="${K8S_CLUSTER_0_NUMBER_OF_NODES}" \
   4
         --machine-type "${K8S_CLUSTER_0_MACHINE_TYPE}" \
   5
         ${GKE_SPOT_INSTANCES_SWITCH:-""}

   1
   gcloud container clusters create "${K8S_CLUSTER_1}" \
   2
         --zone="${K8S_CLUSTER_1_ZONE}" \
   3
         --num-nodes="${K8S_CLUSTER_1_NUMBER_OF_NODES}" \
   4
         --machine-type "${K8S_CLUSTER_1_MACHINE_TYPE}" \
   5
         ${GKE_SPOT_INSTANCES_SWITCH:-""}

   1
   gcloud container clusters create "${K8S_CLUSTER_2}" \
   2
         --zone="${K8S_CLUSTER_2_ZONE}" \
   3
         --num-nodes="${K8S_CLUSTER_2_NUMBER_OF_NODES}" \
   4
         --machine-type "${K8S_CLUSTER_2_MACHINE_TYPE}" \
   5
         ${GKE_SPOT_INSTANCES_SWITCH:-""}

2. 기본값 gcloud 프로젝트 를 설정합니다.

   1
   gcloud config set project "${MDB_GKE_PROJECT}"

3. 자격 증명을 얻고 현재 kubeconfig 파일에 컨텍스트를 저장합니다. 기본적으로 이 파일은 ~/.kube/config 디렉토리에 있으며 $KUBECONFIG 환경 변수에 의해 참조됩니다.

   1
   gcloud container clusters get-credentials "${K8S_CLUSTER_0}" --zone="${K8S_CLUSTER_0_ZONE}"
   2
   gcloud container clusters get-credentials "${K8S_CLUSTER_1}" --zone="${K8S_CLUSTER_1_ZONE}"
   3
   gcloud container clusters get-credentials "${K8S_CLUSTER_2}" --zone="${K8S_CLUSTER_2_ZONE}"

   모든 kubectl 명령은 다음 변수를 사용하여 이러한 컨텍스트를 참조합니다.

   • $K8S_CLUSTER_0_CONTEXT_NAME

   • $K8S_CLUSTER_1_CONTEXT_NAME

   • $K8S_CLUSTER_2_CONTEXT_NAME

4. kubectl 이(가) Kubernetes 클러스터에 액세스할 수 있는지 확인합니다.

   1
   echo "Nodes in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" get nodes
   3
   echo; echo "Nodes in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   4
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" get nodes
   5
   echo; echo "Nodes in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"
   6
   kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" get nodes

   출력 보기

   1
   Nodes in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   2
   NAME                                       STATUS   ROLES    AGE   VERSION
   3
   gke-k8s-mdb-0-default-pool-267f1e8f-d0dg   Ready    <none>   38m   v1.29.7-gke.1104000
   4
   gke-k8s-mdb-0-default-pool-267f1e8f-pmgh   Ready    <none>   38m   v1.29.7-gke.1104000
   5
   gke-k8s-mdb-0-default-pool-267f1e8f-vgj9   Ready    <none>   38m   v1.29.7-gke.1104000
   6
   7
   Nodes in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   8
   NAME                                       STATUS   ROLES    AGE   VERSION
   9
   gke-k8s-mdb-1-default-pool-263d341f-3tbp   Ready    <none>   38m   v1.29.7-gke.1104000
   10
   gke-k8s-mdb-1-default-pool-263d341f-4f26   Ready    <none>   38m   v1.29.7-gke.1104000
   11
   gke-k8s-mdb-1-default-pool-263d341f-z751   Ready    <none>   38m   v1.29.7-gke.1104000
   12
   13
   Nodes in cluster gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
   14
   NAME                                       STATUS   ROLES    AGE   VERSION
   15
   gke-k8s-mdb-2-default-pool-d0da5fd1-chm1   Ready    <none>   38m   v1.29.7-gke.1104000

5. Istio 서비스 메시를 설치하여 Kubernetes 클러스터 간의 클러스터 간 DNS 확인 및 네트워크 연결을 허용합니다.

   1
   CTX_CLUSTER1=${K8S_CLUSTER_0_CONTEXT_NAME} \
   2
   CTX_CLUSTER2=${K8S_CLUSTER_1_CONTEXT_NAME} \
   3
   CTX_CLUSTER3=${K8S_CLUSTER_2_CONTEXT_NAME} \
   4
   ISTIO_VERSION="1.20.2" \
   5
   ../multi-cluster/install_istio_separate_network.sh

네임스페이스를 만듭니다.

• Kubernetes Operator 배포를 위해 $OPERATOR_NAMESPACE 환경 변수 가 참조하는 별도의 네임스페이스 mongodb-operator 을 만듭니다.

• 각 멤버 Kubernetes 클러스터에 동일한 $OPERATOR_NAMESPACE 를 생성합니다. 이는 kubectl MongoDB 플러그인 이 각 멤버 클러스터에서 Kubernetes 연산자에 대한 서비스 계정을 생성할 수 있도록 하기 위해 필요합니다. Kubernetes Operator는 연산자 클러스터에서 이러한 서비스 계정을 사용하여 각 멤버 클러스터에서 작업을 수행합니다.

  1
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
  2
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite
  3
  4
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
  5
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite
  6
  7
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" create namespace "${OPERATOR_NAMESPACE}"
  8
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" label namespace "${OPERATOR_NAMESPACE}" istio-injection=enabled --overwrite

• 연산자 클러스터 역할을 하는 멤버 클러스터를 포함하여 각 멤버 클러스터에서 별도의 네임스페이스인 mongodb 을(를) 만듭니다. Kubernetes Operator는 MongoDB Ops Manager 리소스 및 구성 요소에 이 네임스페이스를 사용합니다.

  1
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" create namespace "${NAMESPACE}"
  2
  kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite
  3
  4
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" create namespace "${NAMESPACE}"
  5
  kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite
  6
  7
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" create namespace "${NAMESPACE}"
  8
  kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" label namespace "${NAMESPACE}" istio-injection=enabled --overwrite

선택 사항. 비공개 이미지 레지스트리에서 시크릿을 가져오도록 클러스터에 권한을 부여합니다.

Quay 레지스트리의 공식 Helm 차트 및 이미지를 사용하는 경우 이 단계는 선택 사항입니다.

1
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" create secret generic "image-registries-secret" \
2
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
3
kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
4
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
5
kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
6
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson
7
kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic "image-registries-secret" \
8
        --from-file=.dockerconfigjson="${HOME}/.docker/config.json" --type=kubernetes.io/dockerconfigjson

선택 사항. 클러스터 연결을 확인합니다.

다음 선택적 스크립트는 서비스 메시가 클러스터 간 DNS 확인 및 연결을 위해 올바르게 구성되었는지 확인합니다.

1. 클러스터 0 에서 이 스크립트를 실행합니다.

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
     apiVersion: apps/v1
   3
     kind: StatefulSet
   4
     metadata:
   5
       name: echoserver0
   6
     spec:
   7
       replicas: 1
   8
       selector:
   9
         matchLabels:
   10
           app: echoserver0
   11
       template:
   12
         metadata:
   13
           labels:
   14
             app: echoserver0
   15
         spec:
   16
           containers:
   17
             - image: k8s.gcr.io/echoserver:1.10
   18
               imagePullPolicy: Always
   19
               name: echoserver0
   20
               ports:
   21
                 - containerPort: 8080
   22
   EOF

2. 클러스터 1 에서 이 스크립트를 실행합니다.

   1
   kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
     apiVersion: apps/v1
   3
     kind: StatefulSet
   4
     metadata:
   5
       name: echoserver1
   6
     spec:
   7
       replicas: 1
   8
       selector:
   9
         matchLabels:
   10
           app: echoserver1
   11
       template:
   12
         metadata:
   13
           labels:
   14
             app: echoserver1
   15
         spec:
   16
           containers:
   17
             - image: k8s.gcr.io/echoserver:1.10
   18
               imagePullPolicy: Always
   19
               name: echoserver1
   20
               ports:
   21
                 - containerPort: 8080
   22
   EOF

3. 클러스터 2 에서 이 스크립트를 실행합니다.

   1
   kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
     apiVersion: apps/v1
   3
     kind: StatefulSet
   4
     metadata:
   5
       name: echoserver2
   6
     spec:
   7
       replicas: 1
   8
       selector:
   9
         matchLabels:
   10
           app: echoserver2
   11
       template:
   12
         metadata:
   13
           labels:
   14
             app: echoserver2
   15
         spec:
   16
           containers:
   17
             - image: k8s.gcr.io/echoserver:1.10
   18
               imagePullPolicy: Always
   19
               name: echoserver2
   20
               ports:
   21
                 - containerPort: 8080
   22
   EOF

4. 이 스크립트를 실행하여 StatefulSet가 생성될 때까지 기다립니다.

   1
   kubectl wait --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver0-0 --timeout=60s
   2
   kubectl wait --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver1-0 --timeout=60s
   3
   kubectl wait --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" --for=condition=ready pod -l statefulset.kubernetes.io/pod-name=echoserver2-0 --timeout=60s

5. 클러스터 0 에 Pod 서비스를 만듭니다.

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver0-0
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       statefulset.kubernetes.io/pod-name: "echoserver0-0"
   13
   EOF

6. 클러스터 1 에 Pod 서비스를 만듭니다.

   1
   kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver1-0
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       statefulset.kubernetes.io/pod-name: "echoserver1-0"
   13
   EOF

7. 클러스터 2 에 Pod 서비스를 만듭니다.

   1
   kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver2-0
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       statefulset.kubernetes.io/pod-name: "echoserver2-0"
   13
   EOF

8. 클러스터 0 에 라운드 로빈 서비스를 생성합니다.

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       app: echoserver0
   13
   EOF

9. 클러스터 1 에 라운드 로빈 서비스를 생성합니다.

   1
   kubectl apply --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: v1
   3
   kind: Service
   4
   metadata:
   5
     name: echoserver
   6
   spec:
   7
     ports:
   8
       - port: 8080
   9
         targetPort: 8080
   10
         protocol: TCP
   11
     selector:
   12
       app: echoserver1
   13
   EOF

10. 클러스터 2 에 라운드 로빈 서비스를 생성합니다.

    1
    kubectl apply --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
    2
    apiVersion: v1
    3
    kind: Service
    4
    metadata:
    5
      name: echoserver
    6
    spec:
    7
      ports:
    8
        - port: 8080
    9
          targetPort: 8080
    10
          protocol: TCP
    11
      selector:
    12
        app: echoserver2
    13
    EOF

11. 클러스터 1 에서 Pod 0 을(를) 확인합니다.

    1
    source_cluster=${K8S_CLUSTER_1_CONTEXT_NAME}
    2
    target_pod="echoserver0-0"
    3
    source_pod="echoserver1-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    출력 보기

    1
    Checking cross-cluster DNS resolution and connectivity from echoserver1-0 in gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1 to echoserver0-0
    2
    SUCCESS

12. 클러스터 0 에서 Pod 1 을(를) 확인합니다.

    1
    source_cluster=${K8S_CLUSTER_0_CONTEXT_NAME}
    2
    target_pod="echoserver1-0"
    3
    source_pod="echoserver0-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    출력 보기

    1
    Checking cross-cluster DNS resolution and connectivity from echoserver0-0 in gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0 to echoserver1-0
    2
    SUCCESS

13. 클러스터 2 에서 Pod 1 을(를) 확인합니다.

    1
    source_cluster=${K8S_CLUSTER_2_CONTEXT_NAME}
    2
    target_pod="echoserver1-0"
    3
    source_pod="echoserver2-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    출력 보기

    1
    Checking cross-cluster DNS resolution and connectivity from echoserver2-0 in gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2 to echoserver1-0
    2
    SUCCESS

15. 클러스터 0 에서 Pod 2 을(를) 확인합니다.

    1
    source_cluster=${K8S_CLUSTER_0_CONTEXT_NAME}
    2
    target_pod="echoserver2-0"
    3
    source_pod="echoserver0-0"
    4
    target_url="http://${target_pod}.${NAMESPACE}.svc.cluster.local:8080"
    5
    echo "Checking cross-cluster DNS resolution and connectivity from ${source_pod} in ${source_cluster} to ${target_pod}"
    6
    out=$(kubectl exec --context "${source_cluster}" -n "${NAMESPACE}" "${source_pod}" -- \
    7
      /bin/bash -c "curl -v ${target_url}" 2>&1);
    8
    grep "Hostname: ${target_pod}" &>/dev/null <<< "${out}" && echo "SUCCESS" || (echo "ERROR: ${out}" && return 1)

    출력 보기

    1
    Checking cross-cluster DNS resolution and connectivity from echoserver0-0 in gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0 to echoserver2-0
    2
    SUCCESS

16. 정리 스크립트를 실행합니다.

    1
    kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver0
    2
    kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver1
    3
    kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete statefulset echoserver2
    4
    kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    5
    kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    6
    kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver
    7
    kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver0-0
    8
    kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver1-0
    9
    kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" delete service echoserver2-0

멀티 클러스터 구성을 배포합니다.

이 단계에서는 kubectl mongodb 플러그인을 사용하여 Kubernetes Operator가 여러 Kubernetes 클러스터에서 워크로드를 관리하는 데 필요한 Kubernetes 클러스터 구성을 자동화합니다.

Kubernetes Operator를 설치하기 전에 Kubernetes 클러스터를 구성하기 때문에 멀티 Kubernetes 클러스터 작업을 위해 Kubernetes Operator를 배포할 때 필요한 모든 멀티 클러스터 구성이 이미 준비되어 있습니다.

개요 에 명시된 대로 Kubernetes Operator에는 MongoDB Ops Manager MongoDB 데이터베이스를 배포하는 데 사용할 수 있는 세 개의 멤버 클러스터에 대한 구성이 있습니다. 첫 번째 클러스터는 Kubernetes 연산자를 설치하고 사용자 지정 리소스를 배포하는 연산자 클러스터로도 사용됩니다.

1
kubectl mongodb multicluster setup \
2
  --central-cluster="${K8S_CLUSTER_0_CONTEXT_NAME}" \
3
  --member-clusters="${K8S_CLUSTER_0_CONTEXT_NAME},${K8S_CLUSTER_1_CONTEXT_NAME},${K8S_CLUSTER_2_CONTEXT_NAME}" \
4
  --member-cluster-namespace="${NAMESPACE}" \
5
  --central-cluster-namespace="${OPERATOR_NAMESPACE}" \
6
  --create-service-account-secrets \
7
  --install-database-roles=true \
8
  --image-pull-secrets=image-registries-secret

1
Ensured namespaces exist in all clusters.
2
creating operator cluster roles in cluster: gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
3
creating member roles in cluster: gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
4
creating member roles in cluster: gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
5
Ensured ServiceAccounts and Roles.
6
Creating KubeConfig secret mongodb-operator/mongodb-enterprise-operator-multi-cluster-kubeconfig in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
7
Ensured database Roles in member clusters.
8
Creating Member list Configmap mongodb-operator/mongodb-enterprise-operator-member-list in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0

Helm 차트를 사용하여 Kubernetes Operator를 설치합니다.

1. MongoDB Helm 리포지토리 를 추가하고 업데이트 합니다. 로컬 Helm 캐시 가 올바른 Kubernetes 연산자 버전을 참조하는지 확인합니다.

   1
   helm repo add mongodb https://mongodb.github.io/helm-charts
   2
   helm repo update mongodb
   3
   helm search repo "${OFFICIAL_OPERATOR_HELM_CHART}"

   출력 보기

   1
   "mongodb" already exists with the same configuration, skipping
   2
   Hang tight while we grab the latest from your chart repositories...
   3
   ...Successfully got an update from the "mongodb" chart repository
   4
   Update Complete. ⎈Happy Helming!⎈
   5
   NAME                       	CHART VERSION	APP VERSION	DESCRIPTION                           
   6
   mongodb/enterprise-operator	1.27.0       	           	MongoDB Kubernetes Enterprise Operator

2. $NAMESPACE 을 감시하고 3개의 멤버 Kubernetes 클러스터를 관리 하도록 구성된 $OPERATOR_NAMESPACE에 Kubernetes Operator를 설치합니다. 절차의 이 점 에서 ServiceAccounts 와 역할 은 kubectl mongodb 플러그인에 의해 이미 배포되었습니다. 따라서 다음 스크립트는 구성을 건너뛰고 operator.createOperatorServiceAccount=false 및 operator.createResourcesServiceAccountsAndRoles=false를 설정하다 . 스크립트는 설정을 지정하여 Helm 차트 multiCluster.clusters 멀티 클러스터 모드에서 Kubernetes 연산자를 배포 하도록 지시합니다.

   1
   helm upgrade --install \
   2
     --debug \
   3
     --kube-context "${K8S_CLUSTER_0_CONTEXT_NAME}" \
   4
     mongodb-enterprise-operator-multi-cluster \
   5
     "${OPERATOR_HELM_CHART}" \
   6
     --namespace="${OPERATOR_NAMESPACE}" \
   7
     --set namespace="${OPERATOR_NAMESPACE}" \
   8
     --set operator.namespace="${OPERATOR_NAMESPACE}" \
   9
     --set operator.watchNamespace="${NAMESPACE}" \
   10
     --set operator.name=mongodb-enterprise-operator-multi-cluster \
   11
     --set operator.createOperatorServiceAccount=false \
   12
     --set operator.createResourcesServiceAccountsAndRoles=false \
   13
     --set "multiCluster.clusters={${K8S_CLUSTER_0_CONTEXT_NAME},${K8S_CLUSTER_1_CONTEXT_NAME},${K8S_CLUSTER_2_CONTEXT_NAME}}" \
   14
     --set "${OPERATOR_ADDITIONAL_HELM_VALUES:-"dummy=value"}"

   출력 보기

   1
   Release "mongodb-enterprise-operator-multi-cluster" does not exist. Installing it now.
   2
   NAME: mongodb-enterprise-operator-multi-cluster
   3
   LAST DEPLOYED: Mon Aug 26 10:55:49 2024
   4
   NAMESPACE: mongodb-operator
   5
   STATUS: deployed
   6
   REVISION: 1
   7
   TEST SUITE: None
   8
   USER-SUPPLIED VALUES:
   9
   dummy: value
   10
   multiCluster:
   11
     clusters:
   12
     - gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   13
     - gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   14
     - gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
   15
   namespace: mongodb-operator
   16
   operator:
   17
     createOperatorServiceAccount: false
   18
     createResourcesServiceAccountsAndRoles: false
   19
     name: mongodb-enterprise-operator-multi-cluster
   20
     namespace: mongodb-operator
   21
     watchNamespace: mongodb
   22
   23
   COMPUTED VALUES:
   24
   agent:
   25
     name: mongodb-agent-ubi
   26
     version: 107.0.0.8502-1
   27
   database:
   28
     name: mongodb-enterprise-database-ubi
   29
     version: 1.27.0
   30
   dummy: value
   31
   initAppDb:
   32
     name: mongodb-enterprise-init-appdb-ubi
   33
     version: 1.27.0
   34
   initDatabase:
   35
     name: mongodb-enterprise-init-database-ubi
   36
     version: 1.27.0
   37
   initOpsManager:
   38
     name: mongodb-enterprise-init-ops-manager-ubi
   39
     version: 1.27.0
   40
   managedSecurityContext: false
   41
   mongodb:
   42
     appdbAssumeOldFormat: false
   43
     imageType: ubi8
   44
     name: mongodb-enterprise-server
   45
     repo: quay.io/mongodb
   46
   mongodbLegacyAppDb:
   47
     name: mongodb-enterprise-appdb-database-ubi
   48
     repo: quay.io/mongodb
   49
   multiCluster:
   50
     clusterClientTimeout: 10
   51
     clusters:
   52
     - gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   53
     - gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   54
     - gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
   55
     kubeConfigSecretName: mongodb-enterprise-operator-multi-cluster-kubeconfig
   56
     performFailOver: true
   57
   namespace: mongodb-operator
   58
   operator:
   59
     additionalArguments: []
   60
     affinity: {}
   61
     createOperatorServiceAccount: false
   62
     createResourcesServiceAccountsAndRoles: false
   63
     deployment_name: mongodb-enterprise-operator
   64
     env: prod
   65
     maxConcurrentReconciles: 1
   66
     mdbDefaultArchitecture: non-static
   67
     name: mongodb-enterprise-operator-multi-cluster
   68
     namespace: mongodb-operator
   69
     nodeSelector: {}
   70
     operator_image_name: mongodb-enterprise-operator-ubi
   71
     replicas: 1
   72
     resources:
   73
       limits:
   74
         cpu: 1100m
   75
         memory: 1Gi
   76
       requests:
   77
         cpu: 500m
   78
         memory: 200Mi
   79
     tolerations: []
   80
     vaultSecretBackend:
   81
       enabled: false
   82
       tlsSecretRef: ""
   83
     version: 1.27.0
   84
     watchNamespace: mongodb
   85
     watchedResources:
   86
     - mongodb
   87
     - opsmanagers
   88
     - mongodbusers
   89
     webhook:
   90
       installClusterRole: true
   91
       registerConfiguration: true
   92
   opsManager:
   93
     name: mongodb-enterprise-ops-manager-ubi
   94
   registry:
   95
     agent: quay.io/mongodb
   96
     appDb: quay.io/mongodb
   97
     database: quay.io/mongodb
   98
     imagePullSecrets: null
   99
     initAppDb: quay.io/mongodb
   100
     initDatabase: quay.io/mongodb
   101
     initOpsManager: quay.io/mongodb
   102
     operator: quay.io/mongodb
   103
     opsManager: quay.io/mongodb
   104
     pullPolicy: Always
   105
   subresourceEnabled: true
   106
   107
   HOOKS:
   108
   MANIFEST:
   109
   ---
   110
   # Source: enterprise-operator/templates/operator-roles.yaml
   111
   kind: ClusterRole
   112
   apiVersion: rbac.authorization.k8s.io/v1
   113
   metadata:
   114
     name: mongodb-enterprise-operator-mongodb-webhook
   115
   rules:
   116
     - apiGroups:
   117
         - "admissionregistration.k8s.io"
   118
       resources:
   119
         - validatingwebhookconfigurations
   120
       verbs:
   121
         - get
   122
         - create
   123
         - update
   124
         - delete
   125
     - apiGroups:
   126
         - ""
   127
       resources:
   128
         - services
   129
       verbs:
   130
         - get
   131
         - list
   132
         - watch
   133
         - create
   134
         - update
   135
         - delete
   136
   ---
   137
   # Source: enterprise-operator/templates/operator-roles.yaml
   138
   kind: ClusterRoleBinding
   139
   apiVersion: rbac.authorization.k8s.io/v1
   140
   metadata:
   141
     name: mongodb-enterprise-operator-multi-cluster-mongodb-operator-webhook-binding
   142
   roleRef:
   143
     apiGroup: rbac.authorization.k8s.io
   144
     kind: ClusterRole
   145
     name: mongodb-enterprise-operator-mongodb-webhook
   146
   subjects:
   147
     - kind: ServiceAccount
   148
       name: mongodb-enterprise-operator-multi-cluster
   149
       namespace: mongodb-operator
   150
   ---
   151
   # Source: enterprise-operator/templates/operator.yaml
   152
   apiVersion: apps/v1
   153
   kind: Deployment
   154
   metadata:
   155
     name: mongodb-enterprise-operator-multi-cluster
   156
     namespace: mongodb-operator
   157
   spec:
   158
     replicas: 1
   159
     selector:
   160
       matchLabels:
   161
         app.kubernetes.io/component: controller
   162
         app.kubernetes.io/name: mongodb-enterprise-operator-multi-cluster
   163
         app.kubernetes.io/instance: mongodb-enterprise-operator-multi-cluster
   164
     template:
   165
       metadata:
   166
         labels:
   167
           app.kubernetes.io/component: controller
   168
           app.kubernetes.io/name: mongodb-enterprise-operator-multi-cluster
   169
           app.kubernetes.io/instance: mongodb-enterprise-operator-multi-cluster
   170
       spec:
   171
         serviceAccountName: mongodb-enterprise-operator-multi-cluster
   172
         securityContext:
   173
           runAsNonRoot: true
   174
           runAsUser: 2000
   175
         containers:
   176
           - name: mongodb-enterprise-operator-multi-cluster
   177
             image: "quay.io/mongodb/mongodb-enterprise-operator-ubi:1.27.0"
   178
             imagePullPolicy: Always
   179
             args:
   180
               - -watch-resource=mongodb
   181
               - -watch-resource=opsmanagers
   182
               - -watch-resource=mongodbusers
   183
               - -watch-resource=mongodbmulticluster
   184
             command:
   185
               - /usr/local/bin/mongodb-enterprise-operator
   186
             volumeMounts:
   187
               - mountPath: /etc/config/kubeconfig
   188
                 name: kube-config-volume
   189
             resources:
   190
               limits:
   191
                 cpu: 1100m
   192
                 memory: 1Gi
   193
               requests:
   194
                 cpu: 500m
   195
                 memory: 200Mi
   196
             env:
   197
               - name: OPERATOR_ENV
   198
                 value: prod
   199
               - name: MDB_DEFAULT_ARCHITECTURE
   200
                 value: non-static
   201
               - name: WATCH_NAMESPACE
   202
                 value: "mongodb"
   203
               - name: NAMESPACE
   204
                 valueFrom:
   205
                   fieldRef:
   206
                     fieldPath: metadata.namespace
   207
               - name: CLUSTER_CLIENT_TIMEOUT
   208
                 value: "10"
   209
               - name: IMAGE_PULL_POLICY
   210
                 value: Always
   211
               # Database
   212
               - name: MONGODB_ENTERPRISE_DATABASE_IMAGE
   213
                 value: quay.io/mongodb/mongodb-enterprise-database-ubi
   214
               - name: INIT_DATABASE_IMAGE_REPOSITORY
   215
                 value: quay.io/mongodb/mongodb-enterprise-init-database-ubi
   216
               - name: INIT_DATABASE_VERSION
   217
                 value: 1.27.0
   218
               - name: DATABASE_VERSION
   219
                 value: 1.27.0
   220
               # Ops Manager
   221
               - name: OPS_MANAGER_IMAGE_REPOSITORY
   222
                 value: quay.io/mongodb/mongodb-enterprise-ops-manager-ubi
   223
               - name: INIT_OPS_MANAGER_IMAGE_REPOSITORY
   224
                 value: quay.io/mongodb/mongodb-enterprise-init-ops-manager-ubi
   225
               - name: INIT_OPS_MANAGER_VERSION
   226
                 value: 1.27.0
   227
               # AppDB
   228
               - name: INIT_APPDB_IMAGE_REPOSITORY
   229
                 value: quay.io/mongodb/mongodb-enterprise-init-appdb-ubi
   230
               - name: INIT_APPDB_VERSION
   231
                 value: 1.27.0
   232
               - name: OPS_MANAGER_IMAGE_PULL_POLICY
   233
                 value: Always
   234
               - name: AGENT_IMAGE
   235
                 value: "quay.io/mongodb/mongodb-agent-ubi:107.0.0.8502-1"
   236
               - name: MDB_AGENT_IMAGE_REPOSITORY
   237
                 value: "quay.io/mongodb/mongodb-agent-ubi"
   238
               - name: MONGODB_IMAGE
   239
                 value: mongodb-enterprise-server
   240
               - name: MONGODB_REPO_URL
   241
                 value: quay.io/mongodb
   242
               - name: MDB_IMAGE_TYPE
   243
                 value: ubi8
   244
               - name: PERFORM_FAILOVER
   245
                 value: 'true'
   246
               - name: MDB_MAX_CONCURRENT_RECONCILES
   247
                 value: "1"
   248
         volumes:
   249
           - name: kube-config-volume
   250
             secret:
   251
               defaultMode: 420
   252
               secretName: mongodb-enterprise-operator-multi-cluster-kubeconfig

3. Kubernetes Operator 배포를 확인합니다.

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" rollout status deployment/mongodb-enterprise-operator-multi-cluster
   2
   echo "Operator deployment in ${OPERATOR_NAMESPACE} namespace"
   3
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" get deployments
   4
   echo; echo "Operator pod in ${OPERATOR_NAMESPACE} namespace"
   5
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${OPERATOR_NAMESPACE}" get pods

   출력 보기

   1
   Waiting for deployment "mongodb-enterprise-operator-multi-cluster" rollout to finish: 0 of 1 updated replicas are available...
   2
   deployment "mongodb-enterprise-operator-multi-cluster" successfully rolled out
   3
   Operator deployment in mongodb-operator namespace
   4
   NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
   5
   mongodb-enterprise-operator-multi-cluster   1/1     1            1           10s
   6
   7
   Operator pod in mongodb-operator namespace
   8
   NAME                                                         READY   STATUS    RESTARTS     AGE
   9
   mongodb-enterprise-operator-multi-cluster-54d786b796-7l5ct   2/2     Running   1 (4s ago)   10s

TLS 인증서를 준비합니다.

이 단계에서는 애플리케이션 데이터베이스 및 MongoDB Ops Manager 애플리케이션에 대해 TLS 를 활성화합니다. TLS를 사용하지 않으려면 MongoDBOpsManager 리소스에서 다음 필드를 제거하세요.

• spec.security.certsSecretPrefix

• spec.security.tls.ca

• spec.applicationDatabase.security.certsSecretPrefix

• spec.applicationDatabase.security.tls.ca

1. 선택 사항. 키 및 인증서를 생성합니다.

   openssl 명령줄 도구를 사용하여 테스트 목적으로 자체 서명된 CA 및 인증서를 생성합니다.

   1
   mkdir certs || true
   2
   3
   cat <<EOF >certs/ca.cnf
   4
   [ req ]
   5
   default_bits       = 2048
   6
   prompt             = no
   7
   default_md         = sha256
   8
   distinguished_name = dn
   9
   x509_extensions    = v3_ca
   10
   11
   [ dn ]
   12
   C=US
   13
   ST=New York
   14
   L=New York
   15
   O=Example Company
   16
   OU=IT Department
   17
   CN=exampleCA
   18
   19
   [ v3_ca ]
   20
   basicConstraints = CA:TRUE
   21
   keyUsage = critical, keyCertSign, cRLSign
   22
   subjectKeyIdentifier = hash
   23
   authorityKeyIdentifier = keyid:always,issuer
   24
   EOF
   25
   26
   cat <<EOF >certs/om.cnf
   27
   [ req ]
   28
   default_bits       = 2048
   29
   prompt             = no
   30
   default_md         = sha256
   31
   distinguished_name = dn
   32
   req_extensions     = req_ext
   33
   34
   [ dn ]
   35
   C=US
   36
   ST=New York
   37
   L=New York
   38
   O=Example Company
   39
   OU=IT Department
   40
   CN=${OPS_MANAGER_EXTERNAL_DOMAIN}
   41
   42
   [ req_ext ]
   43
   subjectAltName = @alt_names
   44
   keyUsage = critical, digitalSignature, keyEncipherment
   45
   extendedKeyUsage = serverAuth, clientAuth
   46
   47
   [ alt_names ]
   48
   DNS.1 = ${OPS_MANAGER_EXTERNAL_DOMAIN}
   49
   DNS.2 = om-svc.${NAMESPACE}.svc.cluster.local
   50
   EOF
   51
   52
   cat <<EOF >certs/appdb.cnf
   53
   [ req ]
   54
   default_bits       = 2048
   55
   prompt             = no
   56
   default_md         = sha256
   57
   distinguished_name = dn
   58
   req_extensions     = req_ext
   59
   60
   [ dn ]
   61
   C=US
   62
   ST=New York
   63
   L=New York
   64
   O=Example Company
   65
   OU=IT Department
   66
   CN=AppDB
   67
   68
   [ req_ext ]
   69
   subjectAltName = @alt_names
   70
   keyUsage = critical, digitalSignature, keyEncipherment
   71
   extendedKeyUsage = serverAuth, clientAuth
   72
   73
   [ alt_names ]
   74
   # multi-cluster mongod hostnames from service for each pod
   75
   DNS.1 = *.${NAMESPACE}.svc.cluster.local
   76
   # single-cluster mongod hostnames from headless service
   77
   DNS.2 = *.om-db-svc.${NAMESPACE}.svc.cluster.local
   78
   EOF
   79
   80
   # generate CA keypair and certificate
   81
   openssl genrsa -out certs/ca.key 2048
   82
   openssl req -x509 -new -nodes -key certs/ca.key -days 1024 -out certs/ca.crt -config certs/ca.cnf
   83
   84
   # generate OpsManager's keypair and certificate
   85
   openssl genrsa -out certs/om.key 2048
   86
   openssl req -new -key certs/om.key -out certs/om.csr -config certs/om.cnf
   87
   88
   # generate AppDB's keypair and certificate
   89
   openssl genrsa -out certs/appdb.key 2048
   90
   openssl req -new -key certs/appdb.key -out certs/appdb.csr -config certs/appdb.cnf
   91
   92
   # generate certificates signed by CA for OpsManager and AppDB
   93
   openssl x509 -req -in certs/om.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/om.crt -days 365 -sha256 -extfile certs/om.cnf -extensions req_ext
   94
   openssl x509 -req -in certs/appdb.csr -CA certs/ca.crt -CAkey certs/ca.key -CAcreateserial -out certs/appdb.crt -days 365 -sha256 -extfile certs/appdb.cnf -extensions req_ext

2. TLS 키로 시크릿을 생성합니다.

   자체 키와 인증서를 사용하려는 경우 이전 생성 단계를 건너뛰고 키와 인증서를 다음 파일에 저장합니다.

   • certs/ca.crt - CA 인증서. 신뢰할 수 있는 인증서를 사용할 때는 필요하지 않습니다.

   • certs/appdb.key - 애플리케이션 데이터베이스의 비공개 키입니다.

   • certs/appdb.crt - 애플리케이션 데이터베이스에 대한 인증서입니다.

   • certs/om.key - MongoDB Ops Manager용 비공개 키입니다.

   • certs/om.crt - MongoDB Ops Manager용 인증서.

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret tls cert-prefix-om-cert \
   2
     --cert=certs/om.crt \
   3
     --key=certs/om.key
   4
   5
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret tls cert-prefix-om-db-cert \
   6
     --cert=certs/appdb.crt \
   7
     --key=certs/appdb.key
   8
   9
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create configmap om-cert-ca --from-file="mms-ca.crt=certs/ca.crt"
   10
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create configmap appdb-cert-ca --from-file="ca-pem=certs/ca.crt"

Ops Manager를 설치합니다.

이제 MongoDB Ops Manager 리소스를 배포하기 위한 환경과 Kubernetes Operator가 준비되었습니다.

1. MongoDB Ops Manager 애플리케이션 인스턴스를 배포한 후 Kubernetes Operator가 생성할 MongoDB Ops Manager 관리자에 필요한 자격 증명을 생성합니다.

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" --namespace "${NAMESPACE}" create secret generic om-admin-user-credentials \
   2
     --from-literal=Username="admin" \
   3
     --from-literal=Password="Passw0rd@" \
   4
     --from-literal=FirstName="Jane" \
   5
     --from-literal=LastName="Doe"

2. 연산자 클러스터라고도 하는 단일 멤버 클러스터에 가능한 가장 간단한 MongoDBOpsManager 사용자 지정 리소스( TLS 가 활성화된 상태)를 배포합니다.

   이 배포서버 는 단일 클러스터 모드 의 배포와 거의 동일하지만 spec.topology 및 spec.applicationDatabase.topology 이(가) MultiCluster 로 설정하다 되어 있습니다.

   이 방법으로 배포하면 단일 Kubernetes 클러스터 배포가 단일 Kubernetes 멤버 클러스터에 대한 다중 Kubernetes 클러스터 배포의 특수한 경우임을 알 수 있습니다. 처음부터 필요한 만큼의 Kubernetes 클러스터에 MongoDB Ops Manager 애플리케이션 및 애플리케이션 데이터베이스 배포를 시작할 수 있으며, 단일 멤버 Kubernetes 클러스터로 배포를 시작할 필요가 없습니다.

   이제 둘 이상의 Kubernetes 클러스터에 걸쳐 있도록 MongoDB Ops Manager 배포를 준비했으며, 이 작업은 이 절차의 뒷부분에서 수행합니다.

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: mongodb.com/v1
   3
   kind: MongoDBOpsManager
   4
   metadata:
   5
     name: om
   6
   spec:
   7
     topology: MultiCluster
   8
     version: "${OPS_MANAGER_VERSION}"
   9
     adminCredentials: om-admin-user-credentials
   10
     security:
   11
       certsSecretPrefix: cert-prefix
   12
       tls:
   13
         ca: om-cert-ca
   14
     clusterSpecList:
   15
       - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   16
         members: 1
   17
     applicationDatabase:
   18
       version: "${APPDB_VERSION}"
   19
       topology: MultiCluster
   20
       security:
   21
         certsSecretPrefix: cert-prefix
   22
         tls:
   23
           ca: appdb-cert-ca
   24
       clusterSpecList:
   25
         - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   26
           members: 3
   27
     backup:
   28
       enabled: false
   29
   EOF

3. Kubernetes Operator가 작업을 시작하고 status.applicationDatabase.phase=Pending 상태에 도달할 때까지 기다립니다. 애플리케이션 데이터베이스와 MongoDB Ops Manager 배포가 모두 완료될 때까지 기다립니다.

   1
   echo "Waiting for Application Database to reach Pending phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s

   출력 보기

   1
   Waiting for Application Database to reach Pending phase...
   2
   mongodbopsmanager.mongodb.com/om condition met

4. MongoDB Ops Manager를 배포합니다. Kubernetes Operator는 다음 단계를 수행하여 MongoDB Ops Manager를 배포합니다. 다음과 같습니다.

   • 애플리케이션 데이터베이스의 복제본 세트 노드를 배포하고 복제본 세트의 MongoDB 프로세스가 실행되기 시작할 때까지 기다립니다.

   • 애플리케이션 데이터베이스의 연결 string 을 사용하여 MongoDB Ops Manager 애플리케이션 인스턴스를 배포하고 준비될 때까지 기다립니다.

   • 각 애플리케이션 데이터베이스의 파드에 모니터링 MongoDB Agent 컨테이너를 추가합니다.

   • MongoDB Ops Manager 애플리케이션과 애플리케이션 데이터베이스 파드가 모두 실행되기를 기다립니다.

   1
   echo "Waiting for Application Database to reach Running phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
   3
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   4
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s
   5
   echo; echo "Waiting for Application Database to reach Pending phase (enabling monitoring)..."
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
   7
   echo "Waiting for Application Database to reach Running phase..."
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=900s
   9
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   10
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=900s
   11
   echo; echo "MongoDBOpsManager resource"
   12
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
   13
   echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   14
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   15
   echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   16
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

   출력 보기

   1
   Waiting for Application Database to reach Running phase...
   2
   mongodbopsmanager.mongodb.com/om condition met
   3
   4
   Waiting for Ops Manager to reach Running phase...
   5
   mongodbopsmanager.mongodb.com/om condition met
   6
   7
   Waiting for Application Database to reach Pending phase (enabling monitoring)...
   8
   mongodbopsmanager.mongodb.com/om condition met
   9
   Waiting for Application Database to reach Running phase...
   10
   mongodbopsmanager.mongodb.com/om condition met
   11
   12
   Waiting for Ops Manager to reach Running phase...
   13
   mongodbopsmanager.mongodb.com/om condition met
   14
   15
   MongoDBOpsManager resource
   16
   NAME   REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE   WARNINGS
   17
   om                7.0.4     Running              Running         Disabled         13m   
   18
   19
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   20
   NAME        READY   STATUS    RESTARTS   AGE
   21
   om-0-0      2/2     Running   0          10m
   22
   om-db-0-0   4/4     Running   0          69s
   23
   om-db-0-1   4/4     Running   0          2m12s
   24
   om-db-0-2   4/4     Running   0          3m32s
   25
   26
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1

   이제 멀티 클러스터 모드 에서 단일 멤버 클러스터를 배포했으므로 둘 이상의 Kubernetes 클러스터에 걸쳐 배포되도록 이 배포를 재구성할 수 있습니다.

5. 두 번째 멤버 클러스터에서 애플리케이션 데이터베이스 복제본 세트 멤버 2개와 MongoDB Ops Manager 애플리케이션의 추가 인스턴스 1개를 배포합니다.

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: mongodb.com/v1
   3
   kind: MongoDBOpsManager
   4
   metadata:
   5
     name: om
   6
   spec:
   7
     topology: MultiCluster
   8
     version: "${OPS_MANAGER_VERSION}"
   9
     adminCredentials: om-admin-user-credentials
   10
     security:
   11
       certsSecretPrefix: cert-prefix
   12
       tls:
   13
         ca: om-cert-ca
   14
     clusterSpecList:
   15
       - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   16
         members: 1
   17
       - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   18
         members: 1
   19
     applicationDatabase:
   20
       version: "${APPDB_VERSION}"
   21
       topology: MultiCluster
   22
       security:
   23
         certsSecretPrefix: cert-prefix
   24
         tls:
   25
           ca: appdb-cert-ca
   26
       clusterSpecList:
   27
         - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   28
           members: 3
   29
         - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   30
           members: 2
   31
     backup:
   32
       enabled: false
   33
   EOF

6. Kubernetes Operator가 작업을 시작할 때까지 기다립니다(보류 단계).

   1
   echo "Waiting for Application Database to reach Pending phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Pending opsmanager/om --timeout=30s

   출력 보기

   1
   Waiting for Application Database to reach Pending phase...
   2
   mongodbopsmanager.mongodb.com/om condition met

7. Kubernetes Operator가 모든 구성 요소 배포를 완료할 때까지 기다립니다.

   1
   echo "Waiting for Application Database to reach Running phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s
   3
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   4
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s
   5
   echo; echo "MongoDBOpsManager resource"
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
   7
   echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   9
   echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   10
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

   출력 보기

   1
   Waiting for Application Database to reach Running phase...
   2
   mongodbopsmanager.mongodb.com/om condition met
   3
   4
   Waiting for Ops Manager to reach Running phase...
   5
   mongodbopsmanager.mongodb.com/om condition met
   6
   7
   MongoDBOpsManager resource
   8
   NAME   REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE   WARNINGS
   9
   om                7.0.4     Running              Running         Disabled         20m   
   10
   11
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   12
   NAME        READY   STATUS    RESTARTS   AGE
   13
   om-0-0      2/2     Running   0          2m56s
   14
   om-db-0-0   4/4     Running   0          7m48s
   15
   om-db-0-1   4/4     Running   0          8m51s
   16
   om-db-0-2   4/4     Running   0          10m
   17
   18
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   19
   NAME        READY   STATUS    RESTARTS   AGE
   20
   om-1-0      2/2     Running   0          3m27s
   21
   om-db-1-0   4/4     Running   0          6m32s
   22
   om-db-1-1   4/4     Running   0          5m5s

백업을 활성화합니다.

MongoDB Ops Manager 애플리케이션의 다중 Kubernetes 클러스터 배포에서는 S3기반 백업 스토리지만 구성할 수 있습니다. 이 절차는 env_variables.sh에 정의된 S3_* 을(를) 참조합니다.

1. 선택 사항. MinIO 연산자를 설치합니다.

   이 절차에서는 MinIO 연산자를 사용하여 백업을 위한 S 호환3저장 배포합니다. Amazon Web Services S3 또는 기타 S3호환 버킷을 사용할 수 있는 경우 이 단계를 건너뛸 수 있습니다. 이 경우 env_variables.sh 에서 S3_* 변수를 적절히 조정합니다.

   1
   kubectl kustomize "github.com/minio/operator/resources/?timeout=120&ref=v5.0.12" | \
   2
     kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
   3
   4
   kubectl kustomize "github.com/minio/operator/examples/kustomization/tenant-tiny?timeout=120&ref=v5.0.12" | \
   5
     kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" apply -f -
   6
   7
   # add two buckets to the tenant config
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "tenant-tiny" patch tenant/myminio \
   9
     --type='json' \
   10
     -p="[{\"op\": \"add\", \"path\": \"/spec/buckets\", \"value\": [{\"name\": \"${S3_OPLOG_BUCKET_NAME}\"}, {\"name\": \"${S3_SNAPSHOT_BUCKET_NAME}\"}]}]"

2. 백업을 구성하고 활성화하기 전에 시크릿을 생성하세요.

   • s3-access-secret - S3 자격 증명을 포함합니다.

   • s3-ca-cert - 버킷의 서버 인증서를 발급한 CA 인증서가 포함되어 있습니다. 이 절차에 사용되는 샘플 MinIO 배포의 경우, 기본 Kubernetes 루트 CA 인증서를 사용하여 인증서에 서명합니다. 공개적으로 신뢰할 수 있는 CA 인증서가 아니므로 MongoDB Ops Manager가 연결을 신뢰할 수 있도록 제공해야 합니다.

   공개적으로 신뢰할 수 있는 인증서를 사용하는 경우 이 단계를 건너뛰고 spec.backup.s3Stores.customCertificateSecretRefs 및 spec.backup.s3OpLogStores.customCertificateSecretRefs 설정에서 값을 제거할 수 있습니다.

   1
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic s3-access-secret \
   2
     --from-literal=accessKey="${S3_ACCESS_KEY}" \
   3
     --from-literal=secretKey="${S3_SECRET_KEY}"
   4
   5
   # minio TLS secrets are signed with the default k8s root CA
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" create secret generic s3-ca-cert \
   7
     --from-literal=ca.crt="$(kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n kube-system get configmap kube-root-ca.crt -o jsonpath="{.data.ca\.crt}")"

백업을 활성화한 상태에서 MongoDB Ops Manager를 다시 배포합니다.

1. Kubernetes Operator는 Kubernetes Operator를 구성하는 모든 멤버 클러스터에서 모든 구성 요소, MongoDB Ops Manager 애플리케이션, 백업 데몬 인스턴스, 애플리케이션 데이터베이스의 복제본 세트 노드를 원하는 조합으로 구성하고 배포할 수 있습니다.

   다중 Kubernetes 클러스터 배포 구성의 유연성을 설명하기 위해 세 번째 멤버 클러스터에 하나의 백업 데몬 인스턴스만 배포하고 첫 번째 및 두 번째 클러스터에 대해 백업 데몬 멤버를 0으로 지정합니다.

   1
   kubectl apply --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" -f - <<EOF
   2
   apiVersion: mongodb.com/v1
   3
   kind: MongoDBOpsManager
   4
   metadata:
   5
     name: om
   6
   spec:
   7
     topology: MultiCluster
   8
     version: "${OPS_MANAGER_VERSION}"
   9
     adminCredentials: om-admin-user-credentials
   10
     security:
   11
       certsSecretPrefix: cert-prefix
   12
       tls:
   13
         ca: om-cert-ca
   14
     clusterSpecList:
   15
       - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   16
         members: 1
   17
         backup:
   18
           members: 0
   19
       - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   20
         members: 1
   21
         backup:
   22
           members: 0
   23
       - clusterName: "${K8S_CLUSTER_2_CONTEXT_NAME}"
   24
         members: 0
   25
         backup:
   26
           members: 1
   27
     configuration: # to avoid configuration wizard on first login
   28
         mms.adminEmailAddr: email@example.com
   29
         mms.fromEmailAddr: email@example.com
   30
         mms.ignoreInitialUiSetup: "true"
   31
         mms.mail.hostname: smtp@example.com
   32
         mms.mail.port: "465"
   33
         mms.mail.ssl: "true"
   34
         mms.mail.transport: smtp
   35
         mms.minimumTLSVersion: TLSv1.2
   36
         mms.replyToEmailAddr: email@example.com
   37
     applicationDatabase:
   38
       version: "${APPDB_VERSION}"
   39
       topology: MultiCluster
   40
       security:
   41
         certsSecretPrefix: cert-prefix
   42
         tls:
   43
           ca: appdb-cert-ca
   44
       clusterSpecList:
   45
         - clusterName: "${K8S_CLUSTER_0_CONTEXT_NAME}"
   46
           members: 3
   47
         - clusterName: "${K8S_CLUSTER_1_CONTEXT_NAME}"
   48
           members: 2
   49
     backup:
   50
       enabled: true
   51
       s3Stores:
   52
         - name: my-s3-block-store
   53
           s3SecretRef:
   54
             name: "s3-access-secret"
   55
           pathStyleAccessEnabled: true
   56
           s3BucketEndpoint: "${S3_ENDPOINT}"
   57
           s3BucketName: "${S3_SNAPSHOT_BUCKET_NAME}"
   58
           customCertificateSecretRefs:
   59
             - name: s3-ca-cert
   60
               key: ca.crt
   61
       s3OpLogStores:
   62
         - name: my-s3-oplog-store
   63
           s3SecretRef:
   64
             name: "s3-access-secret"
   65
           s3BucketEndpoint: "${S3_ENDPOINT}"
   66
           s3BucketName: "${S3_OPLOG_BUCKET_NAME}"
   67
           pathStyleAccessEnabled: true
   68
           customCertificateSecretRefs:
   69
             - name: s3-ca-cert
   70
               key: ca.crt
   71
   EOF

2. Kubernetes Operator 구성이 완료될 때까지 기다립니다.

   1
   echo; echo "Waiting for Backup to reach Running phase..."
   2
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.backup.phase}'=Running opsmanager/om --timeout=1200s
   3
   echo "Waiting for Application Database to reach Running phase..."
   4
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.applicationDatabase.phase}'=Running opsmanager/om --timeout=600s
   5
   echo; echo "Waiting for Ops Manager to reach Running phase..."
   6
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" wait --for=jsonpath='{.status.opsManager.phase}'=Running opsmanager/om --timeout=600s
   7
   echo; echo "MongoDBOpsManager resource"
   8
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get opsmanager/om
   9
   echo; echo "Pods running in cluster ${K8S_CLUSTER_0_CONTEXT_NAME}"
   10
   kubectl --context "${K8S_CLUSTER_0_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   11
   echo; echo "Pods running in cluster ${K8S_CLUSTER_1_CONTEXT_NAME}"
   12
   kubectl --context "${K8S_CLUSTER_1_CONTEXT_NAME}" -n "${NAMESPACE}" get pods
   13
   echo; echo "Pods running in cluster ${K8S_CLUSTER_2_CONTEXT_NAME}"
   14
   kubectl --context "${K8S_CLUSTER_2_CONTEXT_NAME}" -n "${NAMESPACE}" get pods

   출력 보기

   1
   Waiting for Backup to reach Running phase...
   2
   mongodbopsmanager.mongodb.com/om condition met
   3
   Waiting for Application Database to reach Running phase...
   4
   mongodbopsmanager.mongodb.com/om condition met
   5
   6
   Waiting for Ops Manager to reach Running phase...
   7
   mongodbopsmanager.mongodb.com/om condition met
   8
   9
   MongoDBOpsManager resource
   10
   NAME   REPLICAS   VERSION   STATE (OPSMANAGER)   STATE (APPDB)   STATE (BACKUP)   AGE   WARNINGS
   11
   om                7.0.4     Running              Running         Running          26m   
   12
   13
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-a_k8s-mdb-0
   14
   NAME        READY   STATUS    RESTARTS   AGE
   15
   om-0-0      2/2     Running   0          5m11s
   16
   om-db-0-0   4/4     Running   0          13m
   17
   om-db-0-1   4/4     Running   0          14m
   18
   om-db-0-2   4/4     Running   0          16m
   19
   20
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-b_k8s-mdb-1
   21
   NAME        READY   STATUS    RESTARTS   AGE
   22
   om-1-0      2/2     Running   0          5m12s
   23
   om-db-1-0   4/4     Running   0          12m
   24
   om-db-1-1   4/4     Running   0          11m
   25
   26
   Pods running in cluster gke_scratch-kubernetes-team_europe-central2-c_k8s-mdb-2
   27
   NAME                   READY   STATUS    RESTARTS   AGE
   28
   om-2-backup-daemon-0   2/2     Running   0          3m8s

선택 사항. GKE(Google Kubernetes Engine) 클러스터 및 모든 관련 리소스(VM)를 삭제합니다.

다음 스크립트를 실행하여 GKE 클러스터를 삭제하고 환경을 정리합니다.

1
yes | gcloud container clusters delete "${K8S_CLUSTER_0}" --zone="${K8S_CLUSTER_0_ZONE}" &
2
yes | gcloud container clusters delete "${K8S_CLUSTER_1}" --zone="${K8S_CLUSTER_1_ZONE}" &
3
yes | gcloud container clusters delete "${K8S_CLUSTER_2}" --zone="${K8S_CLUSTER_2_ZONE}" &
4
wait