> For the complete MongoDB documentation index, see www.mongodb.com/docs/llms.txt # FAQ: Self-Managed MongoDB Diagnostics This document provides answers to common diagnostic questions and issues. If you don't find the answer you're looking for, check the other FAQ articles or post your question to the [MongoDB Community](https://community.mongodb.com/). ## Where can I find information about a `mongod` process that stopped running unexpectedly? If [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) shuts down unexpectedly on a UNIX or UNIX-based platform, and if [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) fails to log a shutdown or error message, then check your system logs for messages pertaining to MongoDB. For example, for logs located in `/var/log/messages`, use the following commands: ```bash sudo grep mongod /var/log/messages sudo grep score /var/log/messages ``` ## Does TCP `keepalive` time affect MongoDB Deployments? If you experience network timeouts or socket errors in communication between clients and servers, or between members of a sharded cluster or replica set, check the *TCP keepalive value* for the affected systems. Many operating systems set this value to `7200` seconds (two hours) by default. For MongoDB, you will generally experience better results with a shorter keepalive value, on the order of `120` seconds (two minutes). If your MongoDB deployment experiences keepalive-related issues, you must alter the keepalive value on *all* affected systems. This includes all machines running [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) or [`mongos`](https://www.mongodb.com/docs/manual/reference/program/mongos.md#mongodb-binary-bin.mongos) processes and all machines hosting client processes that connect to MongoDB. ### Adjusting the TCP keepalive value: ### Linux - To view the keepalive setting on Linux, use one of the following commands: ```bash sysctl net.ipv4.tcp_keepalive_time ``` Or: ```bash cat /proc/sys/net/ipv4/tcp_keepalive_time ``` The value is measured in seconds. **Note:** Although the setting name includes `ipv4`, the `tcp_keepalive_time` value applies to both IPv4 and IPv6. - To change the `tcp_keepalive_time` value, you can use one of the following commands, supplying a *\* in seconds: ```bash sudo sysctl -w net.ipv4.tcp_keepalive_time= ``` Or: ```bash echo | sudo tee /proc/sys/net/ipv4/tcp_keepalive_time ``` These operations do not persist across system reboots. To persist the setting, add the following line to `/etc/sysctl.conf`, supplying a *\* in seconds, and reboot the machine: ```bash net.ipv4.tcp_keepalive_time = ``` Keepalive values greater than `300` seconds, (5 minutes) will be overridden on [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) and [`mongos`](https://www.mongodb.com/docs/manual/reference/program/mongos.md#mongodb-binary-bin.mongos) sockets and set to `300` seconds. You will need to restart [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) and [`mongos`](https://www.mongodb.com/docs/manual/reference/program/mongos.md#mongodb-binary-bin.mongos) processes for new system-wide keepalive settings to take effect. ## Do TCP Retransmission Timeouts affect MongoDB Deployments? If you experience long stalls (stalls greater than two minutes) followed by network timeouts or socket errors between clients and server or between members of a sharded cluster or replica set, check the `tcp_retries2` value for the affected systems. Most Linux operating systems set this value to `15` by default, while Windows sets it to `5`. For MongoDB, you experience better results with a lower `tcp_retries2` value, on the order of `5` (12 seconds) or lower. If your MongoDB deployment experiences TCP retransmission timeout-related issues, change the `tcp_retries2` value (`TcpMaxDataRetransmission` on Windows) for *all* affected systems. This includes all machines running [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) or [`mongos`](https://www.mongodb.com/docs/manual/reference/program/mongos.md#mongodb-binary-bin.mongos) processes and all machines hosting client processes that connect to MongoDB. ### Adjust the TCP Retransmission Timeout ### Linux On most Linux operating systems, control the TCP retransmission by adjusting the `net.ipv4.tcp_retries2` sysctl setting. **Note:** Although the setting name includes `ipv4`, the `tcp_retries2` setting applies to both IPv4 and IPv6. - To view the current setting, use the `sysctl` command: ```bash sysctl net.ipv4.tcp_retries2 ``` **Output:** ```bash net.ipv4.tcp_retries = 15 ``` - To change the `tcp_retries2` setting at runtime, use the `sysctl` command: ```bash sysctl -w net.ipv4.tcp_retries2=8 ``` - To make the change permanent, edit the configuration file: 1. Open `/etc/sysctl.conf` in your preferred text editor: ```bash vi /etc/sysctl.conf ``` 2. Configure the `net.ipv4.tcp_retries2` setting: ```conf net.ipv4.tcp_retries2 = 8 ``` 3. Restart the system. Your system now uses the new `tcp_retries2` setting. ## Why does MongoDB log so many "Connection Accepted" events? If you see a very large number of connection and re-connection messages in your MongoDB log, then clients are frequently connecting and disconnecting to the MongoDB server. This is normal behavior for applications that do not use request pooling, such as CGI. Consider using FastCGI, an Apache Module, or some other kind of persistent application server to decrease the connection overhead. If these connections do not impact your performance you can use the run-time [`quiet`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-systemLog.quiet) option or the command-line option [`--quiet`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#std-option-mongod.--quiet) to suppress these messages from the log. ## What tools are available for monitoring MongoDB? The [MongoDB Cloud Manager](https://www.mongodb.com/products/tools/cloud-manager) and [Ops Manager, an on-premises solution available in MongoDB Enterprise Advanced](https://www.mongodb.com/products/mongodb-enterprise-advanced) include monitoring functionality, which collects data from running MongoDB deployments and provides visualization and alerts based on that data. For more information, see also the [MongoDB Cloud Manager documentation](https://www.mongodb.com/docs/cloud-manager/) and [Ops Manager documentation.](https://www.mongodb.com/docs/ops-manager/current/application/) A full list of third-party tools is available as part of the [Monitoring a Self-Managed MongoDB Deployment](https://www.mongodb.com/docs/manual/administration/monitoring.md) documentation. ## Memory Diagnostics for the WiredTiger Storage Engine ### Must my working set size fit RAM? No. If the cache does not have enough space to load additional data, WiredTiger evicts pages from the cache to free up space. **Note:** The [`storage.wiredTiger.engineConfig.cacheSizeGB`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizeGB) limits the size of the WiredTiger internal cache. The operating system uses the available free memory for filesystem cache, which allows the compressed MongoDB data files to stay in memory. In addition, the operating system uses any free RAM to buffer file system blocks and file system cache. To accommodate the additional consumers of RAM, you may have to decrease WiredTiger internal cache size. The default WiredTiger internal cache size value assumes that there is a single [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) instance per machine. If a single machine contains multiple MongoDB instances, decrease the setting to accommodate the other [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) instances. If you run [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) in a container (for example, `lxc`, `cgroups`, Docker, etc.) that does *not* have access to all of the RAM available in a system, you must set [`storage.wiredTiger.engineConfig.cacheSizeGB`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizeGB) or [`storage.wiredTiger.engineConfig.cacheSizePct`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizePct) to a value less than the amount of RAM available in the container. The exact amount depends on the other processes running in the container. See [`memLimitMB`.](https://www.mongodb.com/docs/manual/reference/command/hostinfo.md#mongodb-data-hostInfo.system.memLimitMB) You can only provide one of either [`storage.wiredTiger.engineConfig.cacheSizeGB`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizeGB) or [`storage.wiredTiger.engineConfig.cacheSizePct`.](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizePct) To see statistics on the cache and eviction, use the [`serverStatus`](https://www.mongodb.com/docs/manual/reference/command/serverstatus.md#mongodb-dbcommand-dbcmd.serverStatus) command. The [`wiredTiger.cache`](https://www.mongodb.com/docs/manual/reference/command/serverstatus.md#mongodb-serverstatus-serverstatus.wiredTiger.cache) field holds the information on the cache and eviction. ```none ... "wiredTiger" : { ... "cache" : { "tracked dirty bytes in the cache" : , "bytes currently in the cache" : , "maximum bytes configured" : , "bytes read into cache" :, "bytes written from cache" : , "pages evicted by application threads" : , "checkpoint blocked page eviction" : , "unmodified pages evicted" : , "page split during eviction deepened the tree" : , "modified pages evicted" : , "pages selected for eviction unable to be evicted" : , "pages evicted because they exceeded the in-memory maximum" : ,, "pages evicted because they had chains of deleted items" : , "failed eviction of pages that exceeded the in-memory maximum" : , "hazard pointer blocked page eviction" : , "internal pages evicted" : , "maximum page size at eviction" : , "eviction server candidate queue empty when topping up" : , "eviction server candidate queue not empty when topping up" : , "eviction server evicting pages" : , "eviction server populating queue, but not evicting pages" : , "eviction server unable to reach eviction goal" : , "pages split during eviction" : , "pages walked for eviction" : , "eviction worker thread evicting pages" : , "in-memory page splits" : , "percentage overhead" : , "tracked dirty pages in the cache" : , "pages currently held in the cache" : , "pages read into cache" : , "pages written from cache" : , }, ... ``` For an explanation of some key cache and eviction statistics, such as [`wiredTiger.cache.bytes currently in the cache`](https://www.mongodb.com/docs/manual/reference/command/serverstatus.md#mongodb-serverstatus-serverstatus.wiredTiger.cache.bytes-currently-in-the-cache) and [`wiredTiger.cache.tracked dirty bytes in the cache`](https://www.mongodb.com/docs/manual/reference/command/serverstatus.md#mongodb-serverstatus-serverstatus.wiredTiger.cache.tracked-dirty-bytes-in-the-cache), see [`wiredTiger.cache`.](https://www.mongodb.com/docs/manual/reference/command/serverstatus.md#mongodb-serverstatus-serverstatus.wiredTiger.cache) To adjust the size of the WiredTiger internal cache, see [`--wiredTigerCacheSizeGB`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#std-option-mongod.--wiredTigerCacheSizeGB) and [`storage.wiredTiger.engineConfig.cacheSizeGB`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizeGB). Avoid increasing the WiredTiger internal cache size above its default value. If your use case requires increased internal cache size, see [`--wiredTigerCacheSizePct`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#std-option-mongod.--wiredTigerCacheSizePct) and [`storage.wiredTiger.engineConfig.cacheSizePct`.](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizePct) ### How do I calculate how much RAM I need for my application? With WiredTiger, MongoDB utilizes both the WiredTiger internal cache and the filesystem cache. ##### Cache Configuration Settings The default WiredTiger internal cache size is the larger of either: - 50% of (RAM - 1GB), or - 0.256 GB. For example, on a system with a total of 4GB of RAM the WiredTiger cache uses 1.5GB of RAM (`0.5 * (4GB - 1GB) = 1.5 GB`). When providing a specific cache size ensure the RAM does not exceed the bounds of 0.256GB to 10000GB. Avoid increasing the WiredTiger internal cache size above its default value. If your case requires to do so, you can use `--wiredTigerCacheSizePct` to account for changes in memory due to vertical. You must specify a percentage of up to 80% of available memory. Calculated values can range from 0.256GB to 10000GB. For example, on a system with 2GB of RAM the `--wiredTigerCacheSizePct` cannot be set to 10 because 10% of 2GB is 0.2 GB, which is less than 0.256GB. **Note:** In some instances, such as when running in a container that is configured to use less RAM than the amount of memory provisioned for the host, you must account for the limits. You may need to configure the WiredTiger cache to an appropriate value, as WiredTiger may not account for the memory limits of the specific container in certain cases. To view the [`memory limit`](https://www.mongodb.com/docs/manual/reference/command/hostinfo.md#mongodb-data-hostInfo.system.memLimitMB), the value that WiredTiger utilizes as the maximum amount of RAM available use the [`hostInfo`](https://www.mongodb.com/docs/manual/reference/command/hostinfo.md#mongodb-dbcommand-dbcmd.hostInfo) command. By default, WiredTiger uses Snappy block compression for all collections and prefix compression for all indexes. Compression defaults are configurable at a global level and can also be set on a per-collection and per-index basis during collection and index creation. Different representations are used for data in the WiredTiger internal cache versus the on-disk format: - Data in the filesystem cache is the same as the on-disk format, including benefits of any compression for data files. The filesystem cache is used by the operating system to reduce disk I/O. - Indexes loaded in the WiredTiger internal cache have a different data representation to the on-disk format, but can still take advantage of index prefix compression to reduce RAM usage. Index prefix compression deduplicates common prefixes from indexed fields. - Collection data in the WiredTiger internal cache is uncompressed and uses a different representation from the on-disk format. Block compression can provide significant on-disk storage savings, but data must be uncompressed to be manipulated by the server. With the filesystem cache, MongoDB automatically uses all free memory that is not used by the WiredTiger cache or by other processes. To adjust the size of the WiredTiger internal cache, see [`--wiredTigerCacheSizeGB`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#std-option-mongod.--wiredTigerCacheSizeGB) and [`storage.wiredTiger.engineConfig.cacheSizeGB`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizeGB). Avoid increasing the WiredTiger internal cache size above its default value. If your use case requires increased internal cache size, see [`--wiredTigerCacheSizePct`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#std-option-mongod.--wiredTigerCacheSizePct) and [`storage.wiredTiger.engineConfig.cacheSizePct`.](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizePct) **Note:** The [`storage.wiredTiger.engineConfig.cacheSizeGB`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizeGB) limits the size of the WiredTiger internal cache. The operating system uses the available free memory for filesystem cache, which allows the compressed MongoDB data files to stay in memory. In addition, the operating system uses any free RAM to buffer file system blocks and file system cache. To accommodate the additional consumers of RAM, you may have to decrease WiredTiger internal cache size. The default WiredTiger internal cache size value assumes that there is a single [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) instance per machine. If a single machine contains multiple MongoDB instances, decrease the setting to accommodate the other [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) instances. If you run [`mongod`](https://www.mongodb.com/docs/manual/reference/program/mongod.md#mongodb-binary-bin.mongod) in a container (for example, `lxc`, `cgroups`, Docker, etc.) that does *not* have access to all of the RAM available in a system, you must set [`storage.wiredTiger.engineConfig.cacheSizeGB`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizeGB) or [`storage.wiredTiger.engineConfig.cacheSizePct`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizePct) to a value less than the amount of RAM available in the container. The exact amount depends on the other processes running in the container. See [`memLimitMB`.](https://www.mongodb.com/docs/manual/reference/command/hostinfo.md#mongodb-data-hostInfo.system.memLimitMB) You can only provide one of either [`storage.wiredTiger.engineConfig.cacheSizeGB`](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizeGB) or [`storage.wiredTiger.engineConfig.cacheSizePct`.](https://www.mongodb.com/docs/manual/reference/configuration-options.md#mongodb-setting-storage.wiredTiger.engineConfig.cacheSizePct) To view statistics on the cache and eviction rate, see the [`wiredTiger.cache`](https://www.mongodb.com/docs/manual/reference/command/serverstatus.md#mongodb-serverstatus-serverstatus.wiredTiger.cache) field returned from the [`serverStatus`](https://www.mongodb.com/docs/manual/reference/command/serverstatus.md#mongodb-dbcommand-dbcmd.serverStatus) command. ## Sharded Cluster Diagnostics The two most important factors in maintaining a successful sharded cluster are: - [choosing an appropriate shard key](https://www.mongodb.com/docs/manual/core/sharding-shard-key.md#std-label-sharding-internals-shard-keys) and - [sufficient capacity to support current and future operations.](https://www.mongodb.com/docs/manual/core/sharded-cluster-requirements.md#std-label-sharding-capacity-planning) While you can [change your shard key](https://www.mongodb.com/docs/manual/core/sharding-change-a-shard-key.md#std-label-change-a-shard-key) later, it is important to carefully consider your shard key choice to avoid scalability and performance issues. Continue reading for specific issues you may encounter in a production environment. ### In a new sharded cluster, why does all data remain on one shard? Your cluster must have sufficient data for sharding to make sense. Sharding works by migrating chunks between the shards until each shard has roughly the same number of chunks. The default chunk size is 128 megabytes. MongoDB will not begin migrations until the imbalance of chunks in the cluster exceeds the [migration threshold](https://www.mongodb.com/docs/manual/core/sharding-balancer-administration.md#std-label-sharding-migration-thresholds). This behavior helps prevent unnecessary chunk migrations, which can degrade the performance of your cluster as a whole. If you have just deployed a sharded cluster, make sure that you have enough data to make sharding effective. If you do not have sufficient data to create more than eight 128 megabyte chunks, then all data will remain on one shard. Either lower the [chunk size](https://www.mongodb.com/docs/manual/core/sharding-data-partitioning.md#std-label-sharding-chunk-size) setting, or add more data to the cluster. As a related problem, the system will split chunks only on inserts or updates, which means that if you configure sharding and do not continue to issue insert and update operations, the database will not create any chunks. You can either wait until your application inserts data *or* [split chunks manually.](https://www.mongodb.com/docs/manual/tutorial/split-chunks-in-sharded-cluster.md#std-label-split-chunks-sharded-cluster) Finally, if your shard key has a low [cardinality](https://www.mongodb.com/docs/manual/core/sharding-choose-a-shard-key.md#std-label-sharding-shard-key-cardinality), MongoDB may not be able to create sufficient splits among the data. ### Why would one shard receive a disproportionate amount of traffic in a sharded cluster? In some situations, a single shard or a subset of the cluster will receive a disproportionate portion of the traffic and workload. In almost all cases this is the result of a shard key that does not effectively allow [write scaling.](https://www.mongodb.com/docs/manual/sharding.md#std-label-sharding-shard-key-write-scaling) It's also possible that you have "hot chunks." In this case, you may be able to solve the problem by splitting and then migrating parts of these chunks. You may have to consider [resharding your collection](https://www.mongodb.com/docs/manual/core/sharding-reshard-a-collection.md#std-label-sharding-resharding) with a [different shard key](https://www.mongodb.com/docs/manual/core/sharding-choose-a-shard-key.md#std-label-sharding-internals-choose-shard-key) to correct this pattern. ### What can prevent a sharded cluster from balancing? If you have just deployed your sharded cluster, you may want to consider the [troubleshooting suggestions for a new cluster where data remains on a single shard.](https://www.mongodb.com/docs/manual/faq/diagnostics.md#std-label-sharding-troubleshooting-not-splitting) If the cluster was initially balanced, but later developed an uneven distribution of data, consider the following possible causes: - You have deleted or removed a significant amount of data from the cluster. If you have added additional data, it may have a different distribution with regards to its shard key. - Your [shard key](https://www.mongodb.com/docs/manual/reference/glossary.md#std-term-shard-key) has low [cardinality](https://www.mongodb.com/docs/manual/core/sharding-choose-a-shard-key.md#std-label-sharding-shard-key-cardinality) and MongoDB cannot split the chunks any further. - Your data set is growing faster than the balancer can distribute data around the cluster. This is uncommon and typically is the result of: - a [balancing window](https://www.mongodb.com/docs/manual/tutorial/manage-sharded-cluster-balancer.md#std-label-sharding-schedule-balancing-window) that is too short, given the rate of data growth. - an uneven distribution of [write operations](https://www.mongodb.com/docs/manual/sharding.md#std-label-sharding-shard-key-write-scaling) that requires more data migration. You may have to choose a different shard key to resolve this issue. - poor network connectivity between shards, which may lead to chunk migrations that take too long to complete. Investigate your network configuration and interconnections between shards. ### Why do chunk migrations affect sharded cluster performance? If migrations impact your cluster or application's performance, consider the following options, depending on the nature of the impact: 1. If migrations only interrupt your clusters sporadically, you can limit the [balancing window](https://www.mongodb.com/docs/manual/tutorial/manage-sharded-cluster-balancer.md#std-label-sharding-schedule-balancing-window) to prevent balancing activity during peak hours. Ensure that there is enough time remaining to keep the data from becoming out of balance again. 2. If the balancer is always migrating chunks to the detriment of overall cluster performance: - You may want to attempt [decreasing the chunk size](https://www.mongodb.com/docs/manual/tutorial/modify-chunk-size-in-sharded-cluster.md#std-label-tutorial-modifying-chunk-size) to limit the size of the migration. - Your cluster may be over capacity, and you may want to attempt to [add one or two shards](https://www.mongodb.com/docs/manual/tutorial/add-shards-to-shard-cluster.md#std-label-sharding-procedure-add-shard) to the cluster to distribute load. It's also possible that your shard key causes your application to direct all writes to a single shard. This kind of activity pattern can require the balancer to migrate most data soon after writing it. You may have to consider [resharding your collection](https://www.mongodb.com/docs/manual/core/sharding-reshard-a-collection.md#std-label-sharding-resharding) with a [different shard key](https://www.mongodb.com/docs/manual/core/sharding-choose-a-shard-key.md#std-label-sharding-internals-choose-shard-key) that provides better [write scaling.](https://www.mongodb.com/docs/manual/sharding.md#std-label-sharding-shard-key-write-scaling)