TLS NETWORK error in mongo stateful sets

Ops and Admin
1

Hello,

After configuring mongodb as StatefulSets with tls mode = requireTLS I get this NETWORK error:
“c”:“NETWORK”, “id”:22588, “ctx”:“conn527”,“msg”:“Error receiving request from client. Ending connection from remote”,“attr”:{“error”:{“code”:141,“codeName”:“SSLHandshakeFailed”,“errmsg”:“The server is configured to only allow SSL connections”},“remote”:“100.62.8.86:44503”,“connectionId”:527}}
The error seems to come from internal communications between services and StatefulSets.
I don’t make any request from specific client.
With tls mode= preferedTLS it works.
Please advice!

2

Seems something in your K8S (assume you use k8s as you mentioned stateful set) is using non-tls connection to your mongodb server.

You need to figure what that something is and then try configuring it to use tls. (i’m not familiar with stateful set)

1 Like
3

Thank you Kobe.
Indeed I use K8S. MongoDB server is started on TLS config.
The complete message is below:
“c”:“NETWORK”, “id”:22942, “ctx”:“listener”,“msg”:“Connection accepted”,“attr”:{“remote”:“100.62.8.86:44503”,“connectionId”:13,“connectionCount”:1}}
“c”:“NETWORK”, “id”:22986, “ctx”:“conn13”,“msg”:“Error receiving request from client. Ending connection from remote”,“attr”:{“error”:{“code”:141,“codeName”:“SSLHandshakeFailed”,“errmsg”:“The server is configured to only allow SSL connections”},“remote”:“100.62.8.86:44503”,“connectionId”:13}}
“c”:“NETWORK”, “id”:22942, “ctx”:“conn13”,“msg”:“Connection ended”,“attr”:{“remote”:“100.62.8.86:44503” ,“connectionId”:13,“connectionCount”:0}}
I have a service for each replica and a headless service(plus loopback 127.0.0.1). These services communicate (internally) with statefulset and throws these errors.
BR

4

Is this one of the replicas , a service or another connection.

Any connections other then the replica set members will need to transition to tls by adding tls=true to the connection string.

5

Thank you Chris. I knew it. But where should I mention this tls=true? Is there any magic flag? Or should I make a new kind of chart?
My service looks like this:
kind: Service
metadata:
name: myname - internal
spec:
clusterIP: None
ports:
- name: mongodb
port: 27017
selector:
name: myname

6

Hi @Myq

Specifying connection options may be specific to the application, refer to its documentation .

The majority of applications will accept a connections string in the form:
mongodb://username:password@host1,host2,host3/?tls=true

ref:
https://www.mongodb.com/docs/manual/reference/connection-string

7

Yes, but this is available when I make a connection from a specific client (another app as you mentioned) to my mongodb replicaset. In my case the problem occurs when I start mongodb replicaset as a server , because of the internal services communications. Along with these errors, the PODs crashed (scales down).

8

Are you rolling this yourself or using the MongoDB K8S Operator?

The mongod logs / container logs are going to tell you exactly what is going wrong.

9

oc logs mymongo-0 shows the above errors. Is there any logs more precisely?

10

That’s not an error, thats an info level log.

You have stated that it is crashing, there will be logs to indicate why that is the case.

11

These are the only errors I see. Maybe should I switch to debug error level. However seems to be a NETWORK error:
“c”:“NETWORK”, “id”:22986, “ctx”:“conn13”,“msg”:“Error receiving request from client. Ending connection from remote”,“attr”:{“error”:{“code”:141,“codeName”:“SSLHandshakeFailed”,“errmsg”:“The server is configured to only allow SSL connections”},“remote”:“100.62.8.86:44503”,“connectionId”:13}}

12

Again that is not an error. This is logged at "s":"I" an info level log. It is logging the fact a client tried to connect without TLS. It is NOT going to crash a mongod.

You will need to look deeper into your log, if there is a ‘crash’ logs of severity Error or Fatal are common.

Otherwise the container logs may tell you what is going on.

If you’re rolling this yourself what does the healthcheck look like?

ref:
https://www.mongodb.com/docs/manual/reference/log-messages/#structured-logging

13

I rolling by myself. Indeed 100.62.8.86:44503 was another service. I manage to scale down. However now I have the same error from loopback= 127.0.0.1 and the PODs crash after a while.
“s”:“I”, “c”:“NETWORK”, “id”:22988, “ctx”:“conn37”,“msg”:“Error receiving request from client. Ending connection from remote”,“attr”:{“error”:{“code”:141,“codeName”:“SSLHandshakeFailed”,“errmsg”:“The server is configured to only allow SSL connections”},“remote”:“127.0.0.1:54800”
However for a short period I can connect to the database (from a specific client). The TLS/Certificate seems to work fine.
Also I tried to replace bind_ip = 0.0.0.0 with everything else , except localhost/127.0.0.1. The error message disappear but the StatefulSet doesn’t work. I guess localhost/127.0.0.1 is mandatory.

14

What have you configured for your probes?

Are you aware of the MongoDB Community Kubernetes Operator ?

15

Basically I use this readiness probe:
ports:
- containerPort: 27017
readinessProbe:
exec:
command:
- mongo
- --eval
- “db.runCommand({ ping: 1})”
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
failureThreshold: 3

but only for “preferTLS” where everything goes well. For “requireTLS” I have this error:
Readiness probe failed: MongoDB shell version v5 connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb Error: network error while attempting to run command ‘isMaster’ on host ‘127.0.0.1:27017’ : connect@src/mongo/shell/mongo.js:374:17 @(connect):2:6 exception: connect failed exiting with code 1
In this case mongo-1 and mongo-2 do not get to generate.
(I know that MongoDB Community Kubernetes Operator is related only to the cloud. I’m working on prem)

16

What leads you to this conclusion?

If you update to TLS then this would also be required to use TLS. This is why your pod is being stopped(“crashing”), the healthcheck cannot connect.

17

If I use this readiness probe:
readinessProbe:
exec:
command:
- mongo
- --tls
- -u admin
- -p MONGO_ADMIN_PASSWORD
- --tlsCertificateKeyFile=/tlsFolder/mongodb.pem
- --eval
- “db.runCommand({ ping: 1})”
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
failureThreshold: 3
I have this error:
Readiness probe failed: MongoDB shell version connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb {“t”:{“$date”:“2023-11-06T07:22:43.869Z”},“s”:“I”, “c”:“NETWORK”, “id”:5490002, “ctx”:“thread1”,“msg”:“Started a new thread for the timer service”} {“t”:{“$date”:“2023-11-06T07:22:43.879Z”},“s”:“E”, “c”:“NETWORK”, “id”:23256, “ctx”:“js”,“msg”:“SSL peer certificate validation failed”,“attr”:{“error”:“SSL peer certificate validation failed: self signed certificate”}} Error: couldn’t connect to server 127.0.0.1:27017, connection attempt failed: SSLHandshakeFailed: SSL peer certificate validation failed: self signed certificate : connect@src/mongo/shell/mongo.js:374:17 @(connect):2:6 exception: connect failed exiting with code 1
Common Name (CN) is: localhost.
My .pem file is not located in /etc/mongo/ but in other path. Maybe this could be a problem.

18

Where is the CA file for this certificate? You’ll need to specify that on the command line too.

Are you using client certificates in this deployment too or are you confusing options ?

20

Yes seems to be a bug in the certificate.