Starting mongod with encrypted PEM fails, invalid encryption?

Ops and Admin
1

I want to enable SSL authentication on my MongoDB database both at the server and client level.

I obtained a certificate that is signed by my company CA, then created the PEM file as normal by contcatinating the certificate and private key. I did the same for both the server (mongod) and the client (mongo shell) —actually also for Compass client. This all worked just fine.

As a next step, I wanted to encrypt the certificate-key PEM file and use net.ssl.PEMKeyPassword option to start mongod. However, all my attempts so far failed :frowning:.

Here what I tried:

  • I tried to encrypt the whole PEM file => it complaiend from Failed to find PEM blob header: -----BEGIN CERTIFICATE-----.
  • I tried to encrypt the private key alone (then add the result to the PEM file under -----BEGIN RSA PRIVATE KEY-----) => It complained from CryptDecodeObjectEx failed to get size of object: ASN1 bad tag value met.
  • I tried to replace -----BEGIN RSA PRIVATE KEY----- with -----BEGIN ENCRYPTER PRIVATE KEY-----, it complained from Encrypted private keys are not supported, use the Windows certificate store instead.

The encryption command I used is:

openssl rsa -aes256 -in .\mongodb.pem -out mongodb-s.pem

When all the above didn’t work, I suspected the encryption command and used another one:

openssl enc -aes-256-cbc -in .\mongodb.pem -out .\mongodb-s.pem

This one generated a file in a binary format. When I pointed net.ssl.PEMKeyFile to it, it again complained from Failed to find PEM blob header: -----BEGIN CERTIFICATE-----.

For the information, I’m using Windows 10 and starting Mongod v4.2 via Windows Services.

2

In case the question wasn’t clear: do the errors I described refer to a mis-encrypted PEM file? I assume I’m encrypting the correct file PEM (key+cert.).

3

Ok, I have just come to read that PEM encryption doesn’t work on Windows anyway. From the docs.