clientEncryption.Encrypt in C# uses a lot of memory relative to a somewhat large file

We currently have client side encryption setup to encrypt all our sensitive files stored in MongoDB/GridFS. However, we recently encountered a problem where encrypting a file of around 100-200MB will use up more than 2-3GB of memory per file, and this continues to accumulate in RAM usage until the GC is called. Of course, the memory usage isn’t linear, for example for around 100 files, the memory spike is only up to 10-15GB for example. But since we are running in kubernetes/docker, the GC doesn’t seem to know the actual limit of memory where it needs to be called, ending with the pod being deleted due to memory pressure on the node in certain cases.

I’m not sure if we’re doing something wrong, or if there’s a way to limit the memory usage for encrypting those somewhat large files. Below is the code we are using for the encryption:

            var encryptOptions = new EncryptOptions(
                algorithm: EncryptionAlgorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Deterministic.ToString(),
                keyId: dataKeyId);

            var reader = new BinaryReader(stream);
            var data = new BsonBinaryData(reader.ReadBytes((int)stream.Length), BsonBinarySubType.Binary);

            // This is the line  below that causes the memory spike
            var encryptedData = clientEncryption.Encrypt(data, encryptOptions, cancellationToken); 

            using var encryptedStream = new MemoryStream(encryptedData.Bytes);
            base.WriteFileAsync(fileName, fileMetadata, encryptedStream, cancellationToken);

Hi, @PBG,

Thank you for reporting this issue. I have created CSHARP-4669 to investigate further. Please follow CSHARP-4669 in case we have trouble repro’ing the issue or have additional questions.

Sincerely,
James

2 Likes

Hello James, thank you. Will keep an eye on it!

Hello @James_Kovacs, sorry to bother again, but any updates regarding this?

Thank you.

Thanks for reaching out again. We were able to reproduce the issue and have added our analysis to CSHARP-4669. We have some ideas of how to reduce memory usage and will implement them in the coming weeks. You can track progress on CSHARP-4669 as well as reach out to us with any questions or concerns.

Sincerely,
James

1 Like

Great! Thank you James. It’s good to know you were able to find the issue.

Looking forward to the fix. :slight_smile:

Thank you again.

I took a look at the clientEncryption.Encrypt method source code on GitHub and found a possible way to improve the memory management in the MongoDB.Driver.Encryption.ExplicitEncryptionLibMongoCryptController class.

The EncryptField and EncryptFieldAsync methods both call UnwrapValue, which is:

private BsonValue UnwrapValue(byte[] encryptedWrappedBytes)
{
    var rawDocument = new RawBsonDocument(encryptedWrappedBytes);
    return rawDocument["v"];
}

As RawBsonDocument is IDisposable, it should be surrounded by a using clause:

private BsonValue UnwrapValue(byte[] encryptedWrappedBytes)
{
    using var rawDocument = new RawBsonDocument(encryptedWrappedBytes);
    return rawDocument["v"];
}

Not sure whether it could solve the issue, or not, but there may be something to explore further?

Hi, @Pascal,

Reviewing the related code, this appears to be a safe optimization. I’ve created CSHARP-5065 to track this work. You can follow that ticket for updates. Thank you for noting this improvement.

Sincerely,
James

1 Like