Hi, I’m trying to set up CSFLE with AWS KMS using the go driver. As far as I can see, libmongocrypt only accepts setting up credentials by passing access key/secret through the AWS kms provider options.
In our org we use EKS and try to not be setting credentials explicitly but rather use the EKS integration for using roles for pods automatically through Service Accounts, which sets up the AWS_WEB_IDENTITY_TOKEN_FILE in each pod for assuming roles (docs here: Configuring Pods to use a Kubernetes service account - Amazon EKS)
Is there any way for making AWS KMS provider to work with this built into the driver? Or plans for doing that?
In my opinion, it would be great because it can enforce better security practices, and having the driver manage assuming the role and session refreshing would abstract a lot of hassle from the client-side code.
Welcome to the MongoDB Community! In the CSFLE AWS tutorial here, under the Grant Permissions section, there are instructions on how to use an IAM role instead of a user, which will support assume roles. It is in the 2nd yellow box titled “Authenticate with IAM Roles in Production”. I hope that helps.
Thanks for the response Cynthia! I’ve seen that section, but it leaves it to the end user to refresh credentials and recreate clients every time.
If that’s that, I understand the limitation and can share my feedback on it: as a user I’d find it very convenient that the mongo driver used the same environment variables as the AWS SDKs (ideally requiring no passage of credentials if exported in the environment). It would be great if it also handled credentials refreshing (which I think - I haven’t checked all of them - AWS SDK’s already handle automatically when using AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN env vars).
We do have support for automatic refresh, meaning the client doesn’t need to refresh credentials or recreate the clients. It is possible that you are using an older driver version that does not yet support the auto refresh. Are you using the latest version of the driver?
Oh, nice, thanks! could you link me to the docs for setting that up?
I haven’t tried it myself and assumed it didn’t work like that as the documentation you linked in the previous message states:
Your application must include logic to get new temporary credentials and recreate your CSFLE-enabled MongoClient instance when each set of temporary credentials expires.
Thank you for pointing that out. I thought we had removed that from the documentation already and will get that fixed. You should be able to set it up just by using providing the elements listed in the KMS Provider object { "accessKeyId":"<temporary access key ID>", "secretAccessKey":"<temporary secret access key>", "sessionToken":"<temporary session token>" }
Make sure to get the most up to date driver versions to ensure that it has the support. Try it out and let me know how it goes.
Thanks Cynthia, I’ve tried it out as instructed and couldn’t get it to work as I’d understand it should. I’ll be specific about the setup, my understanding about expectations and outcomes.
Ensure latest go mongo driver version (1.12.1)
I’ve set my app with an AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN and use the AWS SDK to retrieve access key/secret and session token from the assumed role.
I use these initial temporary credentials (key + secret + sessiontoken) to configure the
AWS KMS providers for the client’s AutoEncryptOptions. I’m not setting any particular option to indicate “refresh automatically”.
I test the csfle-related workflow upon startup and it works OK.
I assume from what I understood that the driver will refresh the credentials automatically. So…
I test the csfle-related workflow an hour later (after sessions expire by default)
Steps 0-3 work as expected, I can confirm the temporary credentials and CSFLE related workflows work well during the first <60 mins (session expiration), though step 5 fails.
Here is an excerpt from the application logs which reflect the error I’m facing, I’d understand my assumption of refreshing credentials is wrong and that’s not happening.
DEBU[0000] Using mongo driver version: v1.12.1
INFO[0011] 08-11-2023 11:43:05.317336 => GET /configurations/3f488da7-1427-4c8b-acf5-6e2782db2c35/versions/0 from: 127.0.0.1:56262 RequestId=ded44e43-3fdb-4062-a172-02a98b6874f2
DEBU[0012] decoding key: /password owner: name: config-with-secret, version: 0, scope[], labels[] RequestId=ded44e43-3fdb-4062-a172-02a98b6874f2
INFO[2520] 08-11-2023 13:05:43.187757 => GET /configurations/3f488da7-1427-4c8b-acf5-6e2782db2c35/versions/0 from: 127.0.0.1:59922 RequestId=0a229383-bf64-4bdd-93ad-133e56b3642f
DEBU[2521] decoding key: /password owner: name: config-with-secret, version: 0, scope[], labels[] RequestId=0a229383-bf64-4bdd-93ad-133e56b3642f
DEBU[2524] mongo version.Driver: v1.12.1 RequestId=0a229383-bf64-4bdd-93ad-133e56b3642f
DEBU[2524] failed to fetch secret mongocrypt error 1: Error in KMS response. HTTP status=400. Response body=
{"__type":"ExpiredTokenException","message":"The security token included in the request is expired"} <nil> RequestId=0a229383-bf64-4bdd-93ad-133e56b3642f
Perhaps I’m missing some step on how to configure the driver for auto-refreshing?
Thanks
You were correct in your understanding, I just pointed you to the outdated part of the docs. Instead of providing the accesskeyid, secretaccesskey and session token use this instead.
// Passing an empty document results in the driver fetching AWS credentials from the environment.
kmsProviders := map[string]map[string]interface{}{
"aws": {},
}
That tells the driver to fetch new credentials and will not require a restart.
Thank you again for pointing out the outdated text in the docs and let me know how it goes.