`clusterAuthMode: x509` fails when `authorization: enabled`

Hi,
When I enable clusterAuthMode: x509 for replica set authentication, with the proper certs in place per this documentation, the replica members can authenticate with each other just fine. However, when I also enable authorization: enabled, the members are unable to communicate with each other. In the logs I see entries like this:

authenticate db: $external { authenticate: 1, mechanism: "MONGODB-X509", user: "CN=mongod1", $db: "$external" }
2020-09-11T19:15:50.644+0000 I  ACCESS   [conn10] Failed to authenticate CN=mongod1@$external from client 172.19.62.122:37102 with mechanism MONGODB-X509: UserNotFound: Could not find user "CN=mongod1" for db "$external"

This would suggest that a corresponding user needs to be in place for each member of the replica set. In the documentation, it’s clear that a user is needed for clients to use when connecting to the replica set and I do see log entries indicating that my client is successfully authenticating using the user I’ve setup, but I can’t find anything in the docs showing that such a user is needed for internal/membership authorization.

Is such a user needed for each member of the replica set for internal/membership authorization, or am I missing something else here? If a user for each member is needed, what db should the user be created in, and with what permissions/roles; where are the docs I’ve overlooked that specify this?

Thanks for any help.

Hi @DanielP and welcome in the MongoDB Community :muscle: !

Are the cluster IP addresses correctly setup in the bind_ip list?
There are definitely no “internal users” for internal membership authorization, etc. Users are just for real users. Not for internal purposes.

Can you please share your config files & command lines and walk us through the steps you have been through so we can maybe spot the issue?

Cheers,
Maxime.

Appreciate your response!

Here is my conf for the 2 data bearing members:

net:
port: 17017
bindIpAll: true
tls:
mode: requireTLS
CAFile: /certs/myCA.pem
certificateKeyFile: /certs/myCert.pem
replication:
oplogSizeMB: 4096
replSetName: mySet
enableMajorityReadConcern: false
setParameter:
enableLocalhostAuthBypass: true
processManagement:
fork: “false”
storage:
dbPath: /data/db
engine: wiredTiger
journal:
enabled: true
wiredTiger:
collectionConfig:
blockCompressor: snappy
engineConfig:
directoryForIndexes: true
cacheSizeGB: 20
security:
enableEncryption: true
encryptionKeyFile: /etc/mongodb.keyfile
authorization: enabled
clusterAuthMode: x509

And here is my arb conf:

net:
port: 17017
bindIpAll: true
tls:
mode: requireTLS
CAFile: /certs/myCA.pem
certificateKeyFile: /certs/myCert.pem
replication:
oplogSizeMB: 1
replSetName: mySet
enableMajorityReadConcern: false
setParameter:
enableLocalhostAuthBypass: true
processManagement:
fork: “false”
storage:
dbPath: /data/db
engine: wiredTiger
wiredTiger:
collectionConfig:
blockCompressor: snappy
engineConfig:
directoryForIndexes: true
security:
enableEncryption: true
encryptionKeyFile: /etc/mongodb.keyfile
authorization: enabled
clusterAuthMode: x509

When I comment the two lines in the security section, the replica set members can communicate just fine, so it seems to be an issue relating to membership authentication.

Thank you for confirming there are no users needed for internal membership authentication; that was my initial assumption but I’ve been grinding on this problem for several days now so I didn’t want to rule anything out, particularly since the error messages I’m seeing in the logs seem to indicate it’s in fact expecting a particular user named after the member that’s trying to connect.

When you comment security section members are able to communicate means some issue with certificates

From mongo documentation

“…Certificat Requirements: The Distinguished Name (DN), found in the member certificate’s subject, must specify a non-empty value for at least one of the following attributes: Organization (O), the Organizational Unit (OU) or the Domain Component (DC).”

1 Like

I’ve double-checked the certs of the three members; all three have the exact same subject, except for the value of CN; e.g.

$ openssl x509 -in myCert.pem -inform PEM -subject -nameopt RFC2253
subject= emailAddress=dev-team@mycompany.org,CN=mongod1,OU=ops,O=myCompany,L=myCity,ST=myState,C=US

I’ve gone over the documentation you’re referencing several times and I’m meeting all the requirements, namely:

  • All certificates were created from the same CA
  • All certificates contain a non-empty value for at least one of the following: O, OU, or DC
  • All certificates have the exact same DN (excepting the CN value)
  • CN value on each certificate matches the hostname used by the other members
  • extendedKeyUsage is present on all certs and has value clientAuth (TLS Web Client Authentication)

Also, can someone shed light on why I’m seeing errors about:

MONGODB-X509: UserNotFound: Could not find user “CN=mongod1”

when membership authentication is not using users?

some more info:
When I start up the replica set with the following enabled in my conf:

security:
authorization: enabled
clusterAuthMode: x509

And then access the shell on mongod1, authorize w/ the appropriate user, then run rs.status(), I see entries like this for the other members:

{
“_id” : 2,
“name” : “mongod2:17017”,
“health” : 0,
“state” : 8,
“stateStr” : “(not reachable/healthy)”,
“uptime” : 0,
“lastHeartbeat” : ISODate(“2020-09-18T15:37:18.349Z”),
“lastHeartbeatRecv” : ISODate(“1970-01-01T00:00:00Z”),
“pingMs” : NumberLong(0),
“lastHeartbeatMessage” : “Could not find user “CN=mongod1” for db “$external””,
“syncingTo” : “”,
“syncSourceHost” : “”,
“syncSourceId” : -1,
“infoMessage” : “”,
“configVersion” : -1
}

It looks like mongod2 is refusing to communicate with mongod1 because it can’t find the user CN=mongod1. Based on what @MaBeuLux88 stated earlier about users not being needed for membership authentication, should I assume this error message is a red herring? I have no idea why this isn’t working.

per @Ramachandra_Tummala’s comment, I’m further scrutinizing the certs. I don’t know if this is relevant, but I’m creating the certs from an easyrsa CA with the following X509 extensions:

basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
extendedKeyUsage = clientAuth,serverAuth
keyUsage = digitalSignature,keyEncipherment

Anything here that might cause mongod to fail when attempting to use the certificates?

Also, can anyone confirm that error messages like with mechanism MONGODB-X509: UserNotFound: Could not find user "CN=mongod1" for db "$external" can indeed be indicative of problems with the certificates?

Maybe we can work this from the other end; can anyone that is usingclusterAuthMode: x509 with certificates generated with easyrsa3 tell me how they’re doing so, and with what attributes (e.g. which x509-types, etc.)?

Can you extract CN from your certificate with -noout option of openssl
Is it matching with your hostname?

Also in mongodb university course when we implemented x509 i remember an user is created with subject on external db

Please check if this link gives any help

@Ramachandra_Tummala thanks for the response; it’s perhaps important to note that the CN value of the certs is the FQDN of the hosts, not the regular hostname (e.g. hostname is mongod1 whereas the CN value is mongod1.my.domain) Is this perhaps the problem? Would it help to generate a cert that has both the FQDN hostname and the regular hostname in the SAN list?

Also, your comment about needing to add a user, that seems to be solely for client authentication, not membership authentication.

I’m having the exact same problem as yours where replica is returning NoUser error. I have tried both certificates with only clientAuth and both clientAuth and serverAuth

May i know how did you resolve this?

Hello

I did not get this error.I was trying to help another student in the thread you have referred
Please show screenshot of the exact error you got
Check all your certificates are fine .Values of CN,O fields etc
Refer to mongo documentation for more details

It’s okay, I found the error. It was mentioned in the docs but it was all cluttered within sentences so I overlooked.

The Organization attributes ( O ’s), the Organizational Unit attributes ( OU ’s), and the Domain Components ( DC ’s) must match those from both the net.tls.clusterFile and net.tls.certificateKeyFile certificates for the other cluster members (or the tlsX509ClusterAuthDNOverride value, if set).

I had a different certificateKeyFile generated by letsencrypt when I only enabled TLS for client to server communication.