Constantly high CPU usage after OS restart

Ops and Admin
,
1

We have used a mongo v8.0.1 replicaSet (2 nodes + arbiter) on our production servers (Rocky 9.4 with 12 cores) since some weeks. I restartet the whole system of primary after RAM upgrade some days ago.

We have seen 100% cpu usage caused by mongo process on all cores since that.
We have tested folllowing steps:

  • mongo service restart
  • look for slow queries (We also tried to stop all our services calling mongo, but cpu usage won’t change.)
  • node reinitilized
  • switch primary to another node
  • disable transparent huge page on os level

According to mongo, the connection count of node slowly increases overtime. The result of
following statement:

db.currentOp({“secs_running”: {$gte: 3}})

contains a lot of hello commands.

Does anyone has an idea?
Any help would be greatly appreciated

regards

2

Hi @Thomas_Marksteienr

Check out:
https://jira.mongodb.org/browse/SERVER-92749

Downgrading openssl to 3.0 rectifies the issue, per the latest comment it is addressed in an upcoming release:

About the fix - we’ve actually got it working in our master branch and have backported it to 8.0 (it’ll be in the upcoming 8.0.5 release). We’re still working on bringing it back to 7.0 - that should take a few more weeks.

3 Likes
3

hi,
thank you for your answer.

It seems to be our problem. If we disabled tls, there was no problem.

Unfortunately, downgrading openssl is no option for us. Is there any estimation of release date of 8.0.5?

4

after further investigations, we found a way to only change openssl version for mongodb:

cd /tmp
wget http://d.rockylinux.org/vault/rocky/9.4/BaseOS/x86_64/os/Packages/o/openssl-3.0.7-28.el9_4.x86_64.rpm
wget http://d.rockylinux.org/vault/rocky/9.4/BaseOS/x86_64/os/Packages/o/openssl-libs-3.0.7-28.el9_4.x86_64.rpm
mkdir /opt/openssl-3.0.7-28
cd /opt/openssl-3.0.7-28

rpm2cpio /tmp/openssl-3.0.7-28.el9_4.x86_64.rpm | cpio -di
rpm2cpio /tmp/openssl-libs-3.0.7-28.el9_4.x86_64.rpm | cpio -di

set new path in systemd file:

Environment="LD_LIBRARY_PATH=/opt/openssl-3.0.7-28/usr/lib64"

Although the origin problem is gone, we would appreciate a bug fix.

regards

1 Like
5

@Thomas_Marksteienr 8.0.5 is available for download now.