Configure Secret Storage
On this page
You can choose where you store sensitive information for the components that Atlas Kubernetes Operator manages, but Atlas Kubernetes Operator must find the Kubernetes secrets it expects. You can store secrets for Atlas Kubernetes Operator in many ways, including the following methods:
Put sensitive information directly into Kubernetes secrets. All tutorials in the Atlas Kubernetes Operator documentation use Kubernetes secrets by default. To use Kubernetes secrets, follow the steps in the tutorials.
Put sensitive information in a Github repository following a GitOps flow. To store sensitive data in git securely you can use tools, such as Sealed Secrets, which encrypts secrets for the intended target cluster.
Put sensitive information in an external secret storage tool, such as HashiCorp Vault or Hyperscalers native secret management solutions. An intermediary secret provisioning tool fetches sensitive info from the external secret storage tool and creates Kubernetes secrets from the sensitive information. To learn more about the secret provisioning tool, see Considerations.
This tutorial sets up an external secret storage tool for use with Atlas Kubernetes Operator. This tutorial focuses on "secret-less" setups that don't require Atlas Kubernetes Operator to create and store a secret to provision secrets to their Kubernetes cluster.
Considerations
The following tutorial installs or configures the following tools and options:
A secret provisioning tool. The secret provisioning tool uses one or more authentication mechanisms to retrieve the credentials from the secret management service and create secrets that Atlas Kubernetes Operator can use. This tutorial installs one of the following open-source secret provisioning tools:
Authentication to access secrets. You can use different methods to authenticate the service accounts and namespaces that can access secrets in HashiCorp Vault:
For External Secrets Operator, this tutorial uses OIDC JWT authentication. To learn more, see JWT/OIDC authentication.
For Secrets Store CSI Driver, this tutorial uses Kubernetes authentication.
Alternatively, your cloud provider's KMS can use native IAM systems to provide this authentication, which isn't covered in this tutorial. To learn how to configure your cloud provider's KMS for authentication, see the following resources in the External Secrets Operator documentation:
Prerequisites
Before you complete this tutorial, you need the following tools and configurations:
Running service accounts for Kubernetes, Atlas Kubernetes Operator, and Atlas and sufficient privileges to configure them.
You need a running Kubernetes cluster with nodes running processors with the x86-64, AMD64, or ARM64 architecture. For this tutorial, the Kubernetes cluster is
https://kube01.internal.io
listening on the default port (443).You can access the Atlas Kubernetes Operator project on GitHub:
To install the Atlas Kubernetes operator using the Atlas CLI, run the following command:
atlas kubernetes operator install [options] To learn more about the command syntax and parameters, see the Atlas CLI documentation for atlas kubernetes operator install.
To deploy the Atlas Kubernetes Operator, run the following command. Replace
<version>
with the latest release number.kubectl apply -f https://raw.githubusercontent.com/mongodb/mongodb-atlas-kubernetes/<version>/deploy/all-in-one.yaml To register for an Atlas account, see Create an Atlas Account.
API keys. You must create an API key and configure the API Access List.
You need the following public API key, private API key, and the organization ID information to configure Atlas Kubernetes Operator access to Atlas.
If you want Atlas Kubernetes Operator to create a new Atlas project, Create an API (Application Programming Interface) Key in an Organization. If your organization requires an IP access list for the Atlas Administration API, you must also configure the API access list.
Important
You must assign the API key the Organization Project Creator organization role or higher.
If you want to work with an existing Atlas project, Create an API (Application Programming Interface) Key for a Project. If your organization requires an IP access list for the Atlas Administration API, you must also configure the API access list.
Important
You must assign the API key the Project Owner project role.
A secret storage vault. This tutorial uses HashiCorp Vault, which is a third-party service for secret storage, running at
https://vault.internal.io
.You can use other secret storage vaults with Atlas Kubernetes Operator as needed, including Cloud KMS from AWS, Azure, and Google.
Internal access only. To prevent exposing sensitive information over the public internet, the following components of the secret storage solution allow internal access only:
The HashiCorp Vault or KMS service.
The Kubernetes Cluster APIs service.
The internal network. This tutorial uses
internal.io
.
While the previous components allow internal access only, they allow access to each other and allow access to anyone within your team or organization. This is a best practice for security.
Public Certificate Authorities (CAs). You can use public CAs to avoid managing and distributing custom CA root certificates.
You can automate CA cert management and renewal by using any of the following tools:
In this tutorial:
All
internal.io
HTTPs services are internal addresses, but their HTTPS sites hold automatically renewed certificates signed by a public CA.No mutual TLS (mTLS) is required for this integration because it performs only server-side HTTPS validation.
Clients can trust these service certificates without extra certificate provisioning.
Procedure
Follow these steps to configure secret storage for Atlas Kubernetes Operator.
Install the secret provisioning tool in the target cluster.
Select a secret provisioning tool to install it.
To use External Secrets Operator as the secret provisioning tool:
Run the following commands to install External Secrets Operator with Helm Charts and start the service:
helm repo add external-secrets https://charts.external-secrets.io helm upgrade -i --atomic \ -n external-secrets --create-namespace --set extraArgs.loglevel=debug \ external-secrets external-secrets/external-secrets` Ensure External Secrets runs successfully:
kubectl get pod -n external-secrets -l app.kubernetes.io/name=external-secrets NAME READY STATUS RESTARTS AGE external-secrets-5779d5d6f6-2lhgd 1/1 Running 0 70s
To use Secrets Store CSI Driver as the secret provisioning tool, follow these steps:
Run the following command to install Secrets Store CSI Driver with Helm Charts and start the service:
helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts helm upgrade -i --atomic --set syncSecret.enabled=true \ -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver Run the following command to install the Secrets Store CSI HashiCorp Vault plugin with Helm Charts. You don't need to install the HashiCorp Vault server or secrets injector.
helm install vault hashicorp/vault \ --set "server.enabled=false" --set "injector.enabled=false" \ --set "csi.enabled=true" Ensure Secrets Store CSI runs successfully:
kubectl get pod -n kube-system -l app.kubernetes.io/name=secrets-store-csi-driver NAME READY STATUS RESTARTS AGE csi-secrets-store-secrets-store-csi-driver-6dcm8 3/3 Running 0 2m2s Ensure the HashiCorp Vault CSI provider runs successfully:
kubectl get pods -l app.kubernetes.io/name=vault-csi-provider NAME READY STATUS RESTARTS AGE vault-csi-provider-j7xbr 2/2 Running 0 5m39s
Set up authentication to access secrets.
To set up OIDC JWT and Kubernetes authentication:
Run the following command to enable OIDC JWT authentication for the mount path. If you set up several Kubernetes clusters, you must enable OIDC JWT authentication for each cluster's mount path.
vault auth enable -path=jwt-kube01 jwt Run the following command to allow unauthenticated access to the OIDC discovery URLs on the
kube01.internal.io
cluster:$ kubectl create clusterrolebinding oidc-reviewer \ --clusterrole=system:service-account-issuer-discovery \ --group=system:unauthenticated Run the following command to direct HashiCorp Vault to trust the cluster. The issuer for the cluster at
https://kube01.internal.io/
must match the URL in the OIDC discovery document at.well-known/openid-configuration
.curl https://kube01.internal.io/.well-known/openid-configuration | jq .issuer https://kube01.internal.io/ vault write auth/jwt-kube01/config jwks_url="https://kube01.internal.io/openid/v1/jwks" Create the policies that allow access to the secrets you want to expose to the cluster.
The following example creates the
external-secrets
policy that you specify in later steps.echo external-secrets-policy.hcl path "secret/data/kube01/external-secrets/*" { capabilities = ["read"] } Run the command to create a HashiCorp Vault role and bind a Kubernetes service account to the role, which restricts the access of the service account within the HashiCorp Vault.
The following command creates the
jwt-kube01-system
role of the JWT (OIDC) type for thevault
audience. The command specifies thesub
user claim, and thedefault
Kubernetes service account in themongodb-atlas-system
namespace as the bound subject. This command ties the role to theexternal-secrets
policy set of permissions within HashiCorp Vault.vault write auth/jwt-kube01/role/jwt-kube01-system role_type="jwt" bound_audiences=vault \ user_claim="sub" bound_subject="system:serviceaccount:mongodb-atlas-system:default" \ policies="external-secrets" Run the command to create another role for External Secrets Operator to access the
default
namespace.The following command creates the
jwt-kube01-default
role of the JWT (OIDC) type for thevault
audience. This command specifies thesub
user claim, and thedefault
Kubernetes service account in thedefault
namespace as the bound subject . This command ties the role to theexternal-secrets
policy set of permissions within HashiCorp Vault.vault write auth/jwt-kube01/role/jwt-kube01-default role_type="jwt" bound_audiences=vault \ user_claim="sub" bound_subject="system:serviceaccount:default:default" \ policies="external-secrets" Ensure OIDC authentication runs successfully for the system service account:
export TEST_JWT_TOKEN=$(kubectl -n mongodb-atlas-system create token default --audience "vault") vault write auth/jwt-kube01/login role=jwt-kube01-system jwt=$(TEST_JWT_TOKEN) Atlas Kubernetes Operator returns your OIDC JWT credentials for the system service account.
Ensure OIDC authentication runs successfully for the default service account:
export TEST_JWT_TOKEN=$(kubectl -n default create token default --audience "vault") vault write auth/jwt-kube01/login role=jwt-kube01-default jwt=$(TEST_JWT_TOKEN) Atlas Kubernetes Operator returns your OIDC JWT credentials for the default service account.
To set up Kubernetes authentication:
Run the following command to enable Kubernetes authentication for the mount path. If you set up several Kubernetes clusters, you must enable Kubernetes authentication for each cluster's mount path.
vault auth enable -path=jwt-kube01 kubernetes When you install with Helm, it automatically configures cluster role binding.
Create policies that allow access to the secrets you want to expose to the cluster.
The following example creates the policy
vault-secret
used in later steps. It usesvault
, a Kubernetes secret bound to the HashiCorp Vault service account that the Helm Chart CSI provider sets up.echo vault-secret.yaml apiVersion: v1 kind: Secret metadata: name: vault annotations: kubernetes.io/service-account.name: vault type: kubernetes.io/service-account-token $ kubectl apply -f vault-secret.yaml Run the commands to trust the Kubernetes cluster and use Kubernetes authentication:
export VAULT_JWT ?= $(shell kubectl get secret/vault -o jsonpath='{.data.token}' |base64 -d) vault write auth/k8s-kube01/config kubernetes_host="kube01.internal.io" token_reviewer_jwt=$(VAULT_JWT) Run the command to create a HashiCorp Vault role and bind a Kubernetes service account to the role, which restricts the access of the service account within the HashiCorp Vault.
The following command creates the
k8s-kube01
role atauth/k8s-kube01
bound to the Kubernetes service account in thedefault
ormongodb-atlas-system
namespaces. This role ties to thesecrets-store
permissions policy within HashiCorp Vault.vault write auth/k8s-kube01/role/k8s-kube01-role \ bound_service_account_names=default,mongodb-atlas-operator \ bound_service_account_namespaces=default,mongodb-atlas-system \ policies=secrets-store The policies must exist and allow access to the secrets. The following example policy allows access to KV v2 secrets under the
kube01/secrets-store
path:path "secret/data/kube01/secrets-store/*" { capabilities = ["read"] } Ensure Kubernetes authentication runs successfully for the system service account:
export TEST_JWT_TOKEN=$( kubectl create -n mongodb-atlas-system token mongodb-atlas-operator) vault write auth/jwt-kube01/login role=jwt-kube01-system jwt=$(TEST_JWT_TOKEN) Ensure Kubernetes authentication runs successfully for the default account:
export TEST_JWT_TOKEN=$(kubectl -n default create token default) vault write auth/jwt-kube01/login role=jwt-kube01-default jwt=$(TEST_JWT_TOKEN)
Set up automatic secret provisioning.
To provision secrets with External Secrets Operator:
Deploy the
SecretStore
custom resource for thedefault
service account in themongodb-atlas-system
namespace:$ cat external-secrets/vault-system.yaml apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: vault-store namespace: mongodb-atlas-system spec: provider: vault: server: "https://vault.internal.io" path: "secret" version: "v2" auth: jwt: path: "jwt-kube01" role: "jwt-kube01-system" kubernetesServiceAccountToken: expirationSeconds: 600 serviceAccountRef: name: "default" audiences: - vault $ kubectl apply -f external-secrets/vault-system.yaml Deploy the
SecretStore
custom resource for thedefault
service account in thedefault
namespace:$ cat external-secrets/vault-default.yaml apiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: vault-store namespace: default spec: provider: vault: server: "https://vault.internal.io" path: "secret" version: "v2" auth: jwt: path: "jwt-kube01" role: "jwt-role" kubernetesServiceAccountToken: expirationSeconds: 600 serviceAccountRef: name: "default" audiences: - vault $ kubectl apply -f external-secrets/vault-default.yaml Deploy the
ExternalSecret
custom resource that references the secret that contains the API key. You must setspec.target.template.metadata.labels
toatlas.mongodb.com/type
with the valuecredentials
for Atlas Kubernetes Operator to find the secret that External Secrets Operator creates.Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path
secret/data/kube01/external-secrets/atlas-account
with the following properties:orgId
publicApiKey
privateApiKey
$ cat external-secrets/atlas.yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: atlas namespace: mongodb-atlas-system spec: refreshInterval: "15s" secretStoreRef: name: vault-store kind: SecretStore target: name: mongodb-atlas-operator-api-key template: metadata: labels: atlas.mongodb.com/type: credentials data: - secretKey: orgId remoteRef: key: secret/data/kube01/external-secrets/atlas-account property: orgId - secretKey: publicApiKey remoteRef: key: secret/data/kube01/external-secrets/atlas-account property: publicApiKey - secretKey: privateApiKey remoteRef: key: secret/data/kube01/external-secrets/atlas-account property: privateApiKey $ kubectl apply -f external-secrets/atlas.yaml This command creates the Kubernetes secret
mongodb-atlas-operator-api-key
in themongodb-atlas-system
namespace.Deploy the
ExternalSecret
custom resource that references the secret that contains database user credentials. You must setspec.target.template.metadata.labels
toatlas.mongodb.com/type
with the valuecredentials
for Atlas Kubernetes Operator to find the secret that External Secrets Operator creates.Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path
secret/data/kube01/external-secrets/db-user
with thepassword
property.$ cat external-secrets/dbuser.yaml apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: dbuser namespace: default spec: refreshInterval: "15s" secretStoreRef: name: vault-store kind: SecretStore target: name: dbuser-password template: metadata: labels: atlas.mongodb.com/type: credentials data: - secretKey: password remoteRef: key: secret/data/kube01/external-secrets/db-user property: password $ kubectl apply -f external-secrets/atlas.yaml Ensure the secrets return as expected when you run the following commands:
$ kubectl get -n mongodb-atlas-system secrets/mongodb-atlas-operator-api-key $ kubectl get -n default secrets/dbuser-password
To provision secrets with Secrets Store CSI:
Deploy the
SecretProviderClass
custom resource in themongodb-atlas-system
namespace. You must setspec.secretObjects.labels
toatlas.mongodb.com/type
with the valuecredentials
for Atlas Kubernetes Operator to find the secret that Secrets Store CSI creates.Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path
secret/data/kube01/external-secrets/atlas-account
with the following properties:orgId
publicApiKey
privateApiKey
$ cat secrets-store/atlas.yaml apiVersion: secrets-store.csi.x-k8s.io/v1 kind: SecretProviderClass metadata: name: atlas namespace: mongodb-atlas-system spec: provider: vault secretObjects: - data: - key: orgId objectName: atlas-org - key: publicApiKey objectName: atlas-pub-key - key: privateApiKey objectName: atlas-secret-key secretName: mongodb-atlas-operator-api-key type: Opaque labels: atlas.mongodb.com/type: credentials parameters: vaultAddress: https://vault.internal.io vaultKubernetesMountPath: k8s-kube01 roleName: k8s-kube01-role objects: | - objectName: atlas-org secretPath: secret/data/kube01/secrets-store/atlas-account secretKey: orgId - objectName: atlas-pub-key secretPath: secret/data/kube01/secrets-store/atlas-account secretKey: publicApiKey - objectName: atlas-secret-key secretPath: secret/data/kube01/secrets-store/atlas-account secretKey: privateApiKey $ kubectl apply -f secrets-store/atlas.yaml This command creates the Kubernetes secret
mongodb-atlas-operator-api-key
in themongodb-atlas-system
namespace.Run the following command to add a pod that mounts the required secrets:
$ cat secrets-store/ako-patch.yaml template: spec: containers: - name: system-secret-placeholder image: mongodb/atlas command: ["sleep", "infinity"] volumeMounts: - name: secrets-store-mount mountPath: "/mnt/secrets-store" readOnly: true volumes: - name: secrets-store-mount csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: atlas $ kubectl apply -f secrets-store/ako-patch.yaml Deploy the
SecretProviderClass
custom resource that references the secret that contains database user credentials. You must setspec.target.template.metadata.labels
toatlas.mongodb.com/type
with the valuecredentials
for Atlas Kubernetes Operator to find the secret that Secrets Store CSI creates.Before you run the following command, ensure that HashiCorp Vault has the secret populated at the Kv V2 path
secret/data/kube01/external-secrets/db-user
with thepassword
property.$ cat external-secrets/dbuser.yaml apiVersion: external-secrets.io/v1beta1 kind: SecretProviderClass metadata: name: dbuser namespace: default spec: provider: vault secretObjects: - data: - key: password objectName: dbuser secretName: dbuser-password type: Opaque labels: atlas.mongodb.com/type: credentials parameters: vaultAddress: https://vault.internal.io vaultKubernetesMountPath: k8s-kube01 roleName: k8s-kube01-role objects: | - objectName: "dbuser" secretPath: "secret/data/kube01/secrets-store/db-user" secretKey: "password" $ kubectl apply -f secrets-store/dbuser.yaml Run the following command to create the
secret-placeholder
sentinel pod, which ensures the Secrets Store CSI driver fetches thedbuser
credentials and sync them to Kubernetes:$ cat secrets-store/placeholder.yaml kind: Pod apiVersion: v1 metadata: name: secret-placeholder spec: containers: - image: mongodb/atlas command: ["sleep", "infinity"] name: secret-placeholder volumeMounts: - name: secrets-store-mount mountPath: "/mnt/secrets-store" readOnly: true volumes: - name: secrets-store-mount csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: dbuser $ kubectl apply -f secrets-store/placeholder.yaml Ensure the secrets return as expected when you run the following commands:
$ kubectl get -n mongodb-atlas-system secrets/mongodb-atlas-operator-api-key $ kubectl get -n default secrets/dbuser-password
Deploy Atlas Kubernetes Operator custom resources.
You can now deploy Atlas Kubernetes Operator custom resources. Atlas Kubernetes Operator
authenticates with the Kubernetes secrets that reference your
HashiCorp Vault. Adjust the timeout
values as needed for your
deployments.
kubectl apply -f ako/project.yaml kubectl apply -f ako/deployment.yaml kubectl apply -f ako/user.yaml kubectl wait --for=condition=ready atlasdeployment/serverless-deployment --timeout=10m kubectl wait --for=condition=ready atlasdatabaseuser/user --timeout=10m
To learn more about these custom resources, see Custom Resources.
Test your Atlas Kubernetes Operator deployment.
To test your Atlas Kubernetes Operator deployment, run the following command:
export ATLAS_DEPLOYMENT_CONN_STR=$(kubectl get secrets/test-atlas-operator-project-test-serverless-deployment-dbuser -o jsonpath='{.data.connectionStringStandardSrv}' |base64 -d) mongosh $(ATLAS_DEPLOYMENT_CONN_STR) --apiVersion 1 --eval "show dbs"
Atlas Kubernetes Operator returns a list of your database deployments.