MongoDB 4.4.2 / X509 -> Can't connect with mongo shell!

I am trying to set up a stand alone mongod server using X509 and then connect to it from mongo shell. Below is a description of what I do. It all goes well up to a certain point, but when I reach the time to launch mongo shell, troubles show up. Since I am no mongodb expert and neither am I an openssl guru, I may well be making some basic mistake on the way. I hope someone with more experience will take a look and shed some light on the issue.

The root CA certificate (RootCA.pem) is created here, using:

$ openssl req -x509 -newkey rsa:4096 -days 3653 -keyout RootCA.key.pem -out RootCA.pem -subj /C=US/ST=NY/O=RootCA

Then an intermediate certificate (IntermedCA.pem) is created, using:

$ openssl req -config openssl.cnf -new -newkey rsa:4096 -nodes -keyout IntermedCA.key.pem -out IntermedCA.req.pem -subj /C=US/ST=DC/O=IntermedCA
$ openssl x509 -req -days 1096 -in IntermedCA.req.pem -CA RootCA.pem -CAkey RootCA.key.pem -extfile openssl.cnf -extensions v3_ca -set_serial 01 -out IntermedCA.pem

A server certificate (Server.pem) is created, using:

$ openssl req -new -newkey rsa:4096 -nodes -keyout Server.key.pem -out Server.req.pem -subj /C=US/ST=CA/O=ServerCA
$ openssl x509 -req -days 365 -in Server.req.pem -CA IntermedCA.pem -CAkey IntermedCA.key.pem -extensions v3_ca -set_serial 01 -out Server.pem

A client certificate (Client.pem) is created, using:

$ openssl req -new -newkey rsa:4096 -nodes -keyout Client.key.pem -out Client.req.pem -subj /C=US/ST=MA/O=ClientCA
$ openssl x509 -req -days 365 -in Client.req.pem -CA IntermedCA.pem -CAkey IntermedCA.key.pem -extensions v3_ca -set_serial 01 -out Client.pem

The trust chain (TrustChain.pem) for the verification is created, using:

$ cat IntermedCA.pem RootCA.pem > TrustChain.pem

We must then create a certificate of an adequate shape for mongod to work, using:

$ cat Server.key.pem Server.pem > Server.cert

The order of Server.key.pem and Server.pem in the command above does not matter.

The mongod server can then be launched using:

$ mongod --tlsMode requireTLS --tlsCertificateKeyFile Server.cert --tlsCAFile IntermedCA.pem --dbpath /mnt/mongoDB-One/DB_X509 --logpath /mnt/mongoDB-One/DB_X509/mongod.log --fork

Up to this point, all seems to be working perfectly, as far as I can see.

To check, I run:

$ ps -ef | grep mongod
ubuntu 2142 1 1 13:39 ? 00:00:31 mongod --tlsMode requireTLS --tlsCertificateKeyFile Server.cert --tlsCAFile IntermedCA.pem --dbpath /mnt/mongoDB-One/DB_X509 --logpath /mnt/mongoDB-One/DB_X509/mongod.log --fork
ubuntu 2365 2124 0 14:14 pts/0 00:00:00 grep --color=auto mongod
$

and also:

$ sudo netstat -tulpn | grep mongod
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 2142/mongod
$

Then I do for the client the same as I did for the server:

$ cat Client.key.pem Client.pem > Client.cert

And then I try to launch mongo shell, using:

$ mongo --tls --tlsCertificateKeyFile Client.cert --tlsCAFile IntermedCA.pem
MongoDB shell version v4.4.2
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
{“t”:{“$date”:“2021-01-10T14:18:31.198Z”},“s”:“E”, “c”:“NETWORK”, “id”:23256, “ctx”:“js”,“msg”:“SSL peer certificate validation failed”,“attr”:{“error”:“SSL peer certificate validation failed: unable to get issuer certificate”}}
Error: couldn’t connect to server 127.0.0.1:27017, connection attempt failed: SSLHandshakeFailed: SSL peer certificate validation failed: unable to get issuer certificate :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6
exception: connect failed
exiting with code 1
$

As one can see, this last command fails.

And this is the log:

$ tail -3 /mnt/mongoDB-One/DB_X509/mongod.log
{“t”:{“$date”:“2021-01-10T14:18:31.201+00:00”},“s”:“E”, “c”:“NETWORK”, “id”:23256, “ctx”:“conn6”,“msg”:“SSL peer certificate validation failed”,“attr”:{“error”:“SSL peer certificate validation failed: unable to get issuer certificate”}}
{“t”:{“$date”:“2021-01-10T14:18:31.201+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22988, “ctx”:“conn6”,“msg”:“Error receiving request from client. Ending connection from remote”,“attr”:{“error”:{“code”:141,“codeName”:“SSLHandshakeFailed”,“errmsg”:“SSL peer certificate validation failed: unable to get issuer certificate”},“remote”:“127.0.0.1:58650”,“connectionId”:6}}
{“t”:{“$date”:“2021-01-10T14:18:31.201+00:00”},“s”:“I”, “c”:“NETWORK”, “id”:22944, “ctx”:“conn6”,“msg”:“Connection ended”,“attr”:{“remote”:“127.0.0.1:58650”,“connectionId”:6,“connectionCount”:0}}
$

One last detail which may be useful:

Running this command:

$ openssl verify -CAfile TrustChain.pem X

Where X is in the set: {
  Server.pem, Server.cert, Client.pem, Client.cert,
  IntermedCA.pem, RootCA.pem}

always returns:

X: OK

I think you are missing --host in your mongo
For TLS/SSL you have to use --host
The mongo shell verifies the certificate presented by the mongod instance against the specified hostname and the CA file.

So do you mean the command should be this?

mongo --tls --tlsCertificateKeyFile Client.cert --tlsCAFile IntermedCA.pem --host 127.0.0.1

I just tried, it did not work as you can see below:

$ mongo --tls --tlsCertificateKeyFile Client.cert --tlsCAFile IntermedCA.pem --host 127.0.0.1
MongoDB shell version v4.4.2
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
{“t”:{“$date”:“2021-01-12T08:06:01.339Z”},“s”:“E”, “c”:“NETWORK”, “id”:23256, “ctx”:“js”,“msg”:“SSL peer certificate validation failed”,“attr”:{“error”:“SSL peer certificate validation failed: unable to get issuer certificate”}}
Error: couldn’t connect to server 127.0.0.1:27017, connection attempt failed: SSLHandshakeFailed: SSL peer certificate validation failed: unable to get issuer certificate :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6
exception: connect failed
exiting with code 1
$

Please try with FQDN
When you generated root CA certificate what value was given to CN?
CN should match --host in your mongo command

I am not sure what CN is. But in my post (the first one above) you can see how I generate the root CA and tell me if this right or not. I wrote in detail all the commands I used to generate all the certificates, starting with the root CA.

There is only one detail I forgot; I used this command:

$ sed s/CA:FALSE/CA:TRUE/ < /etc/ssl/openssl.cnf > openssl.cnf

to create the openssl.cnf file for the IntermedCA generation.

But this another subject.

The root CA needs to be in the system CAs or specified on the command line.
On ubuntu/debian you can add it to /usr/local/share/ca-certificates/ and execute sudo update-ca-certificates

To specifiy it with mongod use the flag --tlsCAFile

OK, thank you for this tip. As you can see in my first post above, I am using this

--tlsCAFile IntermedCA.pem

on the command line. Is the intermediate certificate not enough in this case?

And trying the other approach, after copying the root CA to /usr/local/share/ca-certificates/ as you suggest, this is what I see running the update-ca-certificates command.

$ sudo update-ca-certificates 
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
$

Is this what is expected?

By searching the net about update-ca-certificates, I have been able to make some slight progress, but it is still not working. I am not aware of what should match what in this whole process, and I am not aware of what is missing when I see messages like:

SSL peer certificate validation failed: unable to get issuer certificate

or:

connection attempt failed: SSLHandshakeFailed: SSL peer certificate validation failed: unable to verify the first certificate

Hi @Michel_Bouchet

No, It needs to be the root of the trust chain.

No. The only file included in the update are ones ending in .crt. Copy the .pem over to a .crt
man 8 update-ca-certificates

Furthermore all certificates with a .crt extension found below /usr/local/share/ca-certificates are also included as implicitly
trusted.


Just reading over the details of your setup steps.

Your server certificate needs the intermediate concatenated to it too. Alternatively add the intermediate to your system’s /usr/local/share/ca-certificates/ too.
cat key.pem cert.pem intermediates.pem > server.pem

OK. I will try, but what do you call key.pem and cert.pem in your last expression ?

They would be the server’s certificate/key pair. I refer to this step from your setup.

So if I am following correctly, you mean that instead of:

$ cat Server.key.pem Server.pem > Server.cert

I need to do:

$ cat Server.key.pem Server.pem IntermedCA.pem > Server.cert

I did as well, without being sure it was needed :

$ cat Client.key.pem Client.pem > Client.cert

Do I also need to do:

$ cat Client.key.pem Client.pem IntermedCA.pem > Client.cert

Is there any condition that I need to match concerning the -subj option parameter when running openssl for the different certificates? Like equality requirement or difference requirement?

After I will try to launch mongod an mongo shell as I am already doing.

Please let me know if you still see a problem.

Yes the client should also have the intermediate.

Check out MongoDB Courses and Trainings | MongoDB University it is a great course on the topic of mongdb security.

Thank you for the tip. I have already completed this course.

But as you must know following a course is one thing. Facing the real world is usually a few steps above :slight_smile:

Now I am trying to polish what I learned, by making some real world use of my knowledge.

Beside, the course is indeed good as you write; but it does not go into details on how you should prepare X509 certificates. They basically come up ready made. This is one thing I had to search from other resources, and as you can see I still have a lot to learn.

… I will try again and see what happens …

Well, even after trying again using this new way; I still get this result when launching mongo shell:

$ mongo --tls --tlsCertificateKeyFile Client.cert --tlsCAFile IntermedCA.pem
MongoDB shell version v4.4.2
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
{"t":{"$date":"2021-01-16T03:23:33.124Z"},"s":"E",  "c":"NETWORK",  "id":23256,   "ctx":"js","msg":"SSL peer certificate validation failed","attr":{"error":"SSL peer certificate validation failed: unable to get issuer certificate"}}
Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SSLHandshakeFailed: SSL peer certificate validation failed: unable to get issuer certificate :
connect@src/mongo/shell/mongo.js:374:17
@(connect):2:6
exception: connect failed
exiting with code 1
$

If the root ca is now installed in the system correctly this should not be needed. Using this it should be the root ca not an intermediate.

So the commands to launch mongod an mongo shell should become this?

mongod --tlsMode requireTLS --tlsCertificateKeyFile Server.cert --tlsCAFile RootCA.pem --dbpath /mnt/mongoDB-One/DB_X509 --logpath /mnt/mongoDB-One/DB_X509/mongod.log --fork

and:

mongo --tls --tlsCertificateKeyFile Client.cert --tlsCAFile RootCA.pem

I may have to double check about the “root ca installed in the system correctly”. Because I am not too confident about that.

But do I need both? I mean “root ca installed in the system” and the option --tlsCAFile above?
Or only one is enough?

… For the time being I have tried running, (starting mongod with --tlsCAFile RootCA.pem) without checking the “root ca installation in the system” :

$ mongo --tls --tlsCertificateKeyFile Client.cert --tlsCAFile RootCA.pem

and it still does not work. I will check the “root ca installation in the system” and see …

It is very likely that one or both certs you are using are not correct.

openssl is a great tool but I find it cumbersome for using as a CA. I would recommend you find a CA Suite you like and use that as a CA. After trying a few I use Hashicorp Vault but that has its own special learning curve.

Another thing, when you run mongo you will want to use a hostname that matches the server’s certificate subject, or one of the SANs, as the next thing will be a handshake error due to a hostname mismatch.
e.g.
The server certificate does not match the host name. Hostname: localhost does not match SAN(s): mongo-0-a, mongo-0-b, mongo-0-c CN: Your Cert CN

I seem to remember having to to it, but courses change and memory is fallible(I did it 2017).

@chris

Indeed there was a mistake and I am now able to connect to the database with mongo shell. I use this command to launch the daemon:

mongod --tlsMode requireTLS --tlsCertificateKeyFile Server.cert --tlsCAFile RootCA.pem --auth --dbpath /mnt/mongoDB-One/DB_X509 --logpath /mnt/mongoDB-One/DB_X509/mongod.log --fork

And then I use this one to fire mongo shell:

mongo --tls --host localhost --tlsCertificateKeyFile Client.cert --tlsCAFile RootCA.pem

Thank you very much for all your help. You certainly gave me a number of valuable advices to reach this point.

I still have to solve a few issues. First check the handling of update-ca-certificates to properly use it. And also though I can set my CN when running openssl to use this to connect:

mongo --tls --host 127.0.0.1 --tlsCertificateKeyFile Client.cert --tlsCAFile RootCA.pem

I am not yet able to use both localhost and 127.0.0.1, and I am not able to use the IP (192.168.1.2) either. I have read that I should make use of subjectAltName, but I haven’t figured out how to do it. What I tried at this point failed.

1 Like