Upgrading from Mongo 5.0.21 community to 6.0.11 community (mongos) fails on RHEL8

I have a simple 3 node sharded-cluster on RHEL8 running 5.0.21 (running without ‘mongos’ issues). I installed using the .tgz distro (for the 6 install/upgrade) and made sure that the files/ports for SELinux were semanaged & restorecon’d and everything was happy accept for ‘mongos’ after I rebooted. Mongos threw the following error:

{“t”:{“$date”:“2023-11-20T22:24:57.717Z”},“s”:“I”, “c”:“NETWORK”, “id”:5693100, “ctx”:“ReplicaSetMonitor-TaskExecutor”,“msg”:“Asio socket.set_option failed with std::system_error”,“attr”:{“note”:“connect (async) TCP fast open”,“option”:{“level”:6,“name”:30,“data”:“01 00 00 00”},“error”:{“what”:“set_option: Operation not supported”,“message”:“Operation not supported”,“category”:“asio.system”,“value”:95}}}$.

I saw others on the forum having similar issues, but I didn’t see anyone experiencing the “value”:95 which the asio boost error code says “operation not supported”. I did run Linux strace on the startup process of mongos and looked specifically for socket errors and other sorts of errors that might throw this asio.system error. I’m not sure what operation it is trying to use that is unsupported. Any thoughts as to what this error could be or relate to?

And just for completeness, here is what my strace on starting the mongos service looked like:

getsockopt(3, SOL_SOCKET, SO_RCVBUF, [212992], [4]) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUFFORCE, [8388608], 4) = 0
getsockopt(3, SOL_SOCKET, SO_SNDBUF, [212992], [4]) = 0
setsockopt(3, SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = 0
connect(3, {sa_family=AF_UNIX, sun_path=“/run/systemd/private”}, 22) = 0
getsockopt(3, SOL_SOCKET, SO_PEERCRED, {pid=1, uid=0, gid=0}, [12]) = 0
getsockopt(3, SOL_SOCKET, SO_PEERSEC, “system_u:system_r:init_t:s0-s0:c”…, [64 => 40]) = 0
getsockopt(3, SOL_SOCKET, SO_PEERGROUPS, 0x55fca2569220, [256 => 0]) = 0
fstat(3, {st_mode=S_IFSOCK|0777, st_size=0, …}) = 0
getsockopt(3, SOL_SOCKET, SO_ACCEPTCONN, [0], [4]) = 0
getsockname(3, {sa_family=AF_UNIX}, [128 => 2]) = 0
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base=“\0AUTH EXTERNAL\r\nDATA\r\n”, iov_len=22}, {iov_base=“NEGOTIATE_UNIX_FD\r\n”, iov_len=19}, {iov_base=“BEGIN\r\n”, iov_len=7}], msg_iovlen=3, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 48
getsockopt(3, SOL_SOCKET, SO_PEERCRED, {pid=1, uid=0, gid=0}, [12]) = 0

And I really can’t put my finger on it as I can’t see anything failing at the socket level.

And if my target system is FIPS compliant, does that mean I really have to set the configurations for mongos and mongod to FIPSMode? (an Enterprise option?).

So, we deactivated all the FIPS/SELinux and suffered the same problem. I believe that the “mongos” executable is the problem on RHEL8 (bug). There is a suspicious ‘\u0000’ at the end of the connections when connecting to my "shardReplSet/mongodb3\u0000 " which doesn’t appear in 5.X. Hmmmm. Also, there isn’t a hidden character in my config, so that’s out. It looks like the \u0000 started to show up at version 6.0.0.

Yeah, so, don’t do this in mongos.conf–> configDB shardReplSet/mongodb1:27019, shardReplSet/mongodb2:27019, shardReplSet/mongodb3:27019. You only need one reference to shardReplSet.

1 Like

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.