How to reencrypt the CSFLE Data Key with the newly rotated Master Key?

PBG · July 18, 2022, 6:57pm

Hello all,

I’ve been researching the requirement to rotate the encryption keys manually. From the KMS (GCP) side, we managed to rotate the master key with the data key still working with the old version of the key.

However, we would like to reencrypt the same data key with the new master key version instead, as from what I saw in this answer, the datakey remained encrypted with the old master key.

I’ve tried to look into the code, and the ClientEncryption class in C# only provides the ability to create new data keys. There is no option to decrypt and reencrypt the CSFLE data key to make sure it’s encrypted with the latest key rotation.

Dmitry_Lukyanov · July 18, 2022, 7:22pm

Hey @PBG , see a coming 2.17 release where we’ve introduced a new rewrapManyDataKey method for this goal into ClientEncryption. It should be released in the next few days

PBG · July 18, 2022, 9:13pm

Oh wow that’s amazing and quite fortunate! Looking forward to it.

Dmitry_Lukyanov · July 18, 2022, 10:07pm

it’s already released Please, check 2.17 release

PBG · July 19, 2022, 11:13am

@Dmitry_Lukyanov Thank you for the quick release!

We’ve ran the rewrap on our test env, and it managed to reencrypt the key, but now the data is inaccessible.
It’s giving the following error:
Encryption related exception: HMAC validation failure.

Please find below the code we used, in case we did something wrong:

    var vaultDatabase = vaultDbClient.GetDatabase(configuration.DatabaseName);

        var keyVaultCollection = vaultDatabase.GetCollection<BsonDocument>(KeyVaultCollectionName);
        var kmsProvider = configuration.KmsProvider ?? throw new InvalidOperationException($"'{nameof(configuration.KmsProvider)}' is expected.");
        var dataKeyDocument = keyVaultCollection.Find($"{{\"masterKey.provider\": \"{kmsProvider}\"}}").SingleOrDefault();
        var dataKey = dataKeyDocument?.GetValue("_id")?.AsNullableGuid;
        FilterDefinition<BsonDocument> filter = dataKeyDocument;
        var dataKeyOptions = GetDataKeyOptions(configuration);
        var rewrapManyDataKeyOptions = new RewrapManyDataKeyOptions(kmsProvider, dataKeyOptions.MasterKey);
        RewrapManyDataKeyResult result = clientEncryption.RewrapManyDataKey(filter, rewrapManyDataKeyOptions);

Dmitry_Lukyanov · July 19, 2022, 3:28pm

Hey, I wasn’t able to reproduce this issue. The gist with code snippet I used: Rewrap · GitHub. Can you please check whether this gist code works for you?
Also, can you please provide the following information:

What is the kmsProvider of the key before and after the call to rewrapManyDataKey ?
The full stacktrace with this error: HMAC validation failure ?

PBG · July 19, 2022, 4:27pm

Thank you for the link. From a first inspection the code seems similar, but I’ll do a deep dive to try and implement the snippet relevant code instead of the one we have.

Regarding your questions:

The KMS provider remains the same in the key vault.

    "masterKey" : {
        "provider" : "gcp",
        "projectId" : "project-example",
        "location" : "location-test",
        "keyRing" : "example-location-keyring",
        "keyName" : "gke-master-key-test"
    }

For the full stacktrace:

 ---> MongoDB.Driver.Encryption.MongoEncryptionException: Encryption related exception: HMAC validation failure.
       ---> MongoDB.Libmongocrypt.CryptException: HMAC validation failure
         at MongoDB.Libmongocrypt.Status.ThrowExceptionIfNeeded()
         at MongoDB.Libmongocrypt.CryptContext.FinalizeForEncryption()
         at MongoDB.Driver.Encryption.LibMongoCryptControllerBase.ProcessStates(CryptContext context, String databaseName, CancellationToken cancellationToken)
         at MongoDB.Driver.Encryption.ExplicitEncryptionLibMongoCryptController.DecryptField(BsonBinaryData encryptedValue, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at MongoDB.Driver.Encryption.ExplicitEncryptionLibMongoCryptController.DecryptField(BsonBinaryData encryptedValue, CancellationToken cancellationToken)
         at PTest.Shared.Database.Mongo.Files.MongoEncryptedFileRepository`1.DecryptStream(ClientEncryption clientEncryption, Stream encryptedStream, CancellationToken cancellationToken) in /home/vsts/work/1/s/shared/dotnet/Database.Mongo/Files/MongoEncryptedFileRepository.cs:line 84
         at PTest.Shared.Database.Mongo.Files.MongoEncryptedFileRepository`1.ReadFileAsync(String fileId, CancellationToken cancellationToken) in /home/vsts/work/1/s/shared/dotnet/Database.Mongo/Files/MongoEncryptedFileRepository.cs:line 43
         at PTest.Shared.Database.Mongo.Files.MongoEncryptedFileRepository`1.ReadFileAsync(String fileId, CancellationToken cancellationToken) in /home/vsts/work/1/s/shared/dotnet/Database.Mongo/Files/MongoEncryptedFileRepository.cs:line 43
         at PTest.Services.Impl.PartIndexing.PartIndexingService.GetPTImageAsync(GetPTImage request, CancellationToken cancellationToken) in /home/vsts/work/1/s/PTest-api/Services/PTest.Services.Impl/PTIndex/PTIndexService.cs:line 478
         --- End of inner exception stack trace ---
         at PTest.Services.Impl.PartIndexing.PartIndexingService.GetPTImageAsync(GetPTImage request, CancellationToken cancellationToken) in /home/vsts/work/1/s/PTest-api/Services/PTest.Services.Impl/PartIndexing/PartIndexingService.cs:line 487
         at PTest.BackendApi.WebApi.Controllers.Public.FilesController.PTImageAsync(Guid pId) in /home/vsts/work/1/s/PTest-api/WebApi/Controllers/Public/FilesController.cs:line 379
         at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()
      --- End of stack trace from previous location ---
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
         at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)
         at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
         at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)
         at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
         at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
         at PTest.BackendApi.WebApi.GlobalExceptionHandler.Invoke(HttpContext context) in /home/vsts/work/1/s/PTest-api/WebApi/GlobalExceptionHandler.cs:line 55

PBG · July 19, 2022, 5:03pm

Alright sorry to be bothering you with this. I’ve implement the //rewrap section of the code, and the issue remains. After looking more closely at the snippet, there is one differences. We’re not using AutoEncrypt. We’re using the MongoFileRepository as a base class to encrypt and decrypt the files stored using GridFS.

        public static Stream EncryptStream(ClientEncryption clientEncryption, Guid dataKeyId, Stream stream, CancellationToken cancellationToken)
        {
            var encryptOptions = new EncryptOptions(
                algorithm: EncryptionAlgorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Deterministic.ToString(),
                keyId: dataKeyId);

            var reader = new BinaryReader(stream);
            var data = new BsonBinaryData(reader.ReadBytes((int)stream.Length));
            var encryptedData = clientEncryption.Encrypt(data, encryptOptions, cancellationToken);
            var encryptedStream = new MemoryStream(encryptedData.Bytes);
            return encryptedStream;
        }

        public static Stream DecryptStream(ClientEncryption clientEncryption, Stream encryptedStream, CancellationToken cancellationToken)
        {
            using var reader = new BinaryReader(encryptedStream);
            var encryptedData = new BsonBinaryData(reader.ReadBytes((int)encryptedStream.Length));
            var data = clientEncryption.Decrypt(encryptedData, cancellationToken);
            var stream = new MemoryStream(data.AsBsonBinaryData.Bytes);
            return stream;
        }

Dmitry_Lukyanov · July 19, 2022, 6:42pm

Can you please provide a full simple self contained repo with grid fs you use?

PBG · July 28, 2022, 4:29pm

@Dmitry_Lukyanov Hello again. I’ve debugged for a few days now, and I managed to pin down what was going wrong. The HMAC bug is happening in a very particular case. There needs to be 3 conditions for it to happen.

Use GridFS to store the encrypted data
Use a KMS (in our case GCP), the bug will not happen when using a local key
Run the rewrap before any encryption or decryption is done with a new ClientEncryption object.

This is the repo as you requested.

As you can see, the only difference between a working rewrap and a non-working one is simply:

            if (shouldTriggerFail)
            {
                Console.WriteLine("Initializing a new client encryption that won't be used to encrypt/decrypt before rewrap");
                clientEncryption = EncryptionProvider.GetClientEncryption(databaseEncryptionConfiguration, vaultClient);
            }

If we use the original clientEncryption object that was initialized for the encryption or decryption, it works perfectly fine. But as soon as we run the rewrap with a yet unused new clientEncryption, the rewrap fails for GridFS.
So for example this works:

var clientEncryption = new ClientEncryption(clientEncryptionOptions);
clientEncryption.Decrypt(encryptedData1, CancellationToken.None);
encryptionProvider.RewrapKeys(databaseEncryptionConfiguration, vaultClient, clientEncryption);
clientEncryption.Decrypt(encryptedData2, CancellationToken.None);

But this will fail:

var clientEncryption = new ClientEncryption(clientEncryptionOptions);
encryptionProvider.RewrapKeys(databaseEncryptionConfiguration, vaultClient, clientEncryption);
clientEncryption.Decrypt(encryptedData1, CancellationToken.None);
clientEncryption.Decrypt(encryptedData2, CancellationToken.None);

Dmitry_Lukyanov · July 28, 2022, 5:04pm

Thanks @PBG , Thanks for your repo, I can confirm that I see the same behavior, we will look at it and come back to you.

Dmitry_Lukyanov · July 29, 2022, 12:50pm

@PBG , it’s a bug, we’re working on fixing it. You can track related work here

PBG · July 29, 2022, 1:03pm

@Dmitry_Lukyanov Thank you for the confirmation. Best of luck!

Dmitry_Lukyanov · July 30, 2022, 1:09am

@PBG , please check release 2.17.1 that contains solution for this bug

PBG · July 30, 2022, 5:18pm

@Dmitry_Lukyanov That works! Thank you!

system · August 4, 2022, 5:18pm

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.