Hello! We’re currently building out an application using MongoDB Atlas (M30) that takes advantage of Explicit Encryption through CSFLE. When encrypting a single object before insertion using the deterministic algorithm, it takes about 0.05 seconds to encrypt each key in that object. Assuming that there was 30 keys for every object we needed to encrypt, we’re talking about ~1.5 seconds for encryption on that object (we opted for the deterministic algorithm rather than random since we will need to query on a large portion of the 30 different keys). This is totally fine with a good portion of our api calls that only insert or update one document at a time but there are some instances where we will be encrypting and inserting ~70,000 documents at a time. Assuming all of these would also have 30 keys per object the math works out to (70000 objects x .05 sec per key) x 30 keys which gives us ~105,000 seconds (just over 29 hours) for the encryption of that single insertion. Assuming we only encrypted a single key in those 70,000 objects, we would still be looking at ~70,000 x .05 sec (just under an hour). Is this performance expected when using Explicit Encryption on large amounts of data? Or might we potentially be doing something wrong? I’ve attached added some basic logic below to show what we’re doing below, thank you!
from pymongo import MongoClient
from pymongo.encryption import ClientEncryption
from pymongo.encryption_options import AutoEncryptionOpts
from pymongo.encryption import Algorithm
import base64
import bson
# kms providers object
kms_providers = {
"EXAMPLE": "PLACEHOLDER"
}
# AutoEncryptionOpts values
fle_opts = AutoEncryptionOpts(
kms_providers=kms_providers,
key_vault_namespace="KEY_VAULT_NAMESPACE_HERE",
bypass_auto_encryption=True
)
# create connection
connection = MongoClient(
"CONNECTION_STRING_HERE",
auto_encryption_opts=fle_opts
)
# create helper functions that will be used to encrypt the data
def clientEncryption(collection):
return ClientEncryption(
kms_providers,
"ENCRYPTION_KEYVAULT_NAMESPACE_HERE",
connection,
collection.codec_options
)
def encryptData(data, collection):
res = clientEncryption(collection).encrypt(
data,
Algorithm.AEAD_AES_256_CBC_HMAC_SHA_512_Deterministic,
key_id = bson.Binary(base64.b64decode("KEY_ID_HERE"), "SUBTYPE_INT_HERE"),
)
return res
client = connection["TEST_DATABASE_NAME"]
collection = client.testCollection
insert_data = []
# create 1000 test dictionaries with 30 keys each
for i in range(0, 1000):
insert_data_dict = {
"encrypted_data1": encryptData('Test String', collection),
"encrypted_data2": encryptData('Test String', collection),
"encrypted_data3": encryptData('Test String', collection),
"encrypted_data4": encryptData('Test String', collection),
"encrypted_data5": encryptData('Test String', collection),
"encrypted_data6": encryptData('Test String', collection),
"encrypted_data7": encryptData('Test String', collection),
"encrypted_data8": encryptData('Test String', collection),
"encrypted_data9": encryptData('Test String', collection),
"encrypted_data10": encryptData('Test String', collection),
"encrypted_data11": encryptData('Test String', collection),
"encrypted_data12": encryptData('Test String', collection),
"encrypted_data13": encryptData('Test String', collection),
"encrypted_data14": encryptData('Test String', collection),
"encrypted_data15": encryptData('Test String', collection),
"encrypted_data16": encryptData('Test String', collection),
"encrypted_data17": encryptData('Test String', collection),
"encrypted_data18": encryptData('Test String', collection),
"encrypted_data19": encryptData('Test String', collection),
"encrypted_data20": encryptData('Test String', collection),
"encrypted_data21": encryptData('Test String', collection),
"encrypted_data22": encryptData('Test String', collection),
"encrypted_data23": encryptData('Test String', collection),
"encrypted_data24": encryptData('Test String', collection),
"encrypted_data25": encryptData('Test String', collection),
"encrypted_data26": encryptData('Test String', collection),
"encrypted_data27": encryptData('Test String', collection),
"encrypted_data28": encryptData('Test String', collection),
"encrypted_data29": encryptData('Test String', collection),
"encrypted_data30": encryptData('Test String', collection),
}
insert_data.append(insert_data_dict.copy())
# insert the data
collection.insert_many(insert_data)