TCMalloc Performance Optimization for a Self-Managed Deployment
Changed in version 8.0.
Starting in MongoDB 8.0, MongoDB uses an upgraded version of TCMalloc that uses per-CPU caches, instead of per-thread caches, to reduce memory fragmentation and make your database more resilient to high-stress workloads.
To use the new TCMalloc version:
Platform Support
Operating systems that support MongoDB 8.0 also support the updated TCMalloc, except for the following:
RHEL 8 / CentOS 8 / Oracle 8 on the PPC64LE and s390x architectures
RHEL 9 / CentOS 9 / Oracle 9 on the PPC64LE architecture
These operating systems use the legacy TCMalloc version. If you use these operating systems, disable THP.
Enable Transparent Hugepages (THP)
Transparent Hugepages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups. THP achieves this by combining small pages and making them appear as larger memory pages to the application.
In MongoDB 8.0 and later, ensure that THP is enabled before
mongod
starts by creating a service file for your
platform's initialization system. If you are using
MongoDB 7.0 or earlier, disable THP.
Note
You can enable THP at the system level and disable it on the process level. If you have multiple MongoDB processes on a single machine, ensure that processes on version 8.0 enable THP, while processes on 7.0 or earlier disable THP.
Additionally, for RHEL and CentOS systems
that use ktune
and tuned
performance profiles, you must also create a
custom tuned
profile.
Create a Service File
To create a service file that enables THP, use the built-in initialization
system for your platform. Recent versions of Linux typically use systemd,
which uses the systemctl
command. Older versions of Linux tend to
use System V init, which uses the service
command. For more information,
see the documentation for your operating system.
Use the initialization system for your platform:
Create the systemd
unit file
Create the following file and save it at
/etc/systemd/system/enable-transparent-huge-pages.service
:
[Unit] Description=Enable Transparent Hugepages (THP) DefaultDependencies=no After=sysinit.target local-fs.target Before=mongod.service [Service] Type=oneshot ExecStart=/bin/sh -c 'echo always | tee /sys/kernel/mm/transparent_hugepage/enabled > /dev/null && echo defer+madvise | tee /sys/kernel/mm/transparent_hugepage/defrag > /dev/null && echo 0 | tee /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none > /dev/null && echo 1 | tee /proc/sys/vm/overcommit_memory > /dev/null' [Install] WantedBy=basic.target
Note
Some versions of Red Hat Enterprise Linux, and potentially other Red
Hat-based derivatives, use a different path for the THP enabled
file:
/sys/kernel/mm/redhat_transparent_hugepage/enabled
Verify which path is in use on your system and update the
enable-transparent-huge-pages.service
file accordingly.
Start the service
Run:
sudo systemctl start enable-transparent-huge-pages
To verify that the relevant THP settings have changed, run the following command:
cat /sys/kernel/mm/transparent_hugepage/enabled && cat /sys/kernel/mm/transparent_hugepage/defrag && cat /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none && cat /proc/sys/vm/overcommit_memory
On Red Hat Enterprise Linux and potentially other Red Hat-based derivatives, you may instead need to use the following:
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled && cat /sys/kernel/mm/redhat_transparent_hugepage/defrag && cat /sys/kernel/mm/redhat_transparent_hugepage/khugepaged/max_ptes_none && cat /proc/sys/vm/overcommit_memory
The output should resemble the following:
always defer+madvise 0 1
Create the init.d
script
Create the following file and save it at
/etc/init.d/enable-transparent-hugepages
:
!/bin/bash ## BEGIN INIT INFO Provides: enable-transparent-hugepages Required-Start: $local_fs Required-Stop: X-Start-Before: mongod mongodb-mms-automation-agent Default-Start: 2 3 4 5 Default-Stop: 0 1 6 Short-Description: Enable Linux Transparent Hugepages Description: Enable Linux Transparent Hugepages, to improve database performance. ## END INIT INFO case $1 in start) if [ -d /sys/kernel/mm/transparent_hugepage ]; then thp_path=/sys/kernel/mm/transparent_hugepage elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then thp_path=/sys/kernel/mm/redhat_transparent_hugepage else return 0 fi echo 'always' | tee ${thp_path}/enabled > /dev/null && echo defer+madvise | tee ${thp_path}/defrag > /dev/null && echo 0 | tee ${thp_path}/khugepaged/max_ptes_none > /dev/null && echo 1 | tee /proc/sys/vm/overcommit_memory > /dev/null' unset thp_path ;; esac
Run the script
Run:
sudo /etc/init.d/enable-transparent-hugepages start
To verify that the relevant THP settings have changed, run the following command:
cat /sys/kernel/mm/transparent_hugepage/enabled && cat /sys/kernel/mm/transparent_hugepage/defrag && cat /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none && cat /proc/sys/vm/overcommit_memory
On Red Hat Enterprise Linux and potentially other Red Hat-based derivatives, you may instead need to use the following:
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
The output should resemble the following:
always defer+madvise 0 1
Configure your operating system to run it on boot
To ensure that this setting is applied each time the operating sytem starts, run the following command for your Linux distribution:
Distribution | Command | |
---|---|---|
Ubuntu and Debian |
| |
SUSE |
| |
Red Hat, CentOS, Amazon Linux, and derivatives |
|
Using tuned
and ktune
Important
If you use tuned
or ktune
, perform the steps in this section after
creating the service file.
tuned
and ktune
are kernel tuning utilities that can affect
the Transparent Hugepages setting on your system. If you are using
tuned
or ktune
on your RHEL
or CentOS system while running mongod
, you must create a custom
tuned
profile to ensure that THP stays enabled.
Enable Per-CPU Caches
To verify that TCMalloc is running with per-CPU caches, ensure that:
tcmalloc.usingPerCPUCaches
istrue
.tcmalloc.tcmalloc.cpu_free
is greater than0
.
If per-CPU caches aren't enabled, ensure that:
You disable glibc rseq.
You're using Linux kernel version 4.18 or later.
Disable glibc rseq
The new TCMalloc requires Restartable Sequences (rseq) to implement per-CPU caches. If another application, such as the glibc library, registers an rseq structure before TCMalloc, TCMalloc can't use rseq. Without rseq, TCMalloc uses per-thread caches, which are used by the legacy TCMalloc version.
To ensure that TCMalloc can use rseq to enable per-CPU caches, you can
disable glibc’s registration of a rseq structure. To disable glibc rseq, set
the following environment variable before you start mongod
:
GLIBC_TUNABLES=glibc.pthread.rseq=0 export GLIBC_TUNABLES
Check Kernel Version
If you disabled glibc rseq and per-CPU caches are still not enabled, ensure that you're using Linux kernel version 4.18 or later. To check your kernel version, run the following command:
uname -r