System-Initiated Query Exceeding Time Limit on Secondary Server

Summary

We are observing system-initiated queries on our MongoDB secondary server that exceed the maximum time limit (MaxTimeMSExpired). These queries are performed by the __system@local user.

Environment

  • MongoDB Version: 5.0.18

Problem Description

We have identified queries in the system.profile collection that are initiated by the __system@local user. These queries are targeting the mp.test1 collection and are exceeding the specified maxTimeMS limit. Below is an example of one such document from the system.profile collection:

{
    "op" : "query",
    "ns" : "mp.test1",
    "command" : {
        "find" : "test1",
        "filter" : {
            "detailsId" : "668dcf000000000000000000"
        },
        "batchSize" : 1.0,
        "singleBatch" : true,
        "maxTimeMS" : 1000.0,
        "$readPreference" : {
            "mode" : "secondaryPreferred"
        },
        "readConcern" : {
            "level" : "local"
        },
        "$db" : "mp"
    },
    "numYield" : 402.0,
    "queryHash" : "75B43167",
    "planCacheKey" : "9D7CG57B",
    "locks" : {
        "FeatureCompatibilityVersion" : {
            "acquireCount" : {
                "r" : NumberLong(404)
            }
        },
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(403)
            }
        },
        "Mutex" : {
            "acquireCount" : {
                "r" : NumberLong(1)
            }
        }
    },
    "flowControl" : {

    },
    "readConcern" : {
        "level" : "local",
        "provenance" : "clientSupplied"
    },
    "ok" : 0.0,
    "errMsg" : "Executor error during find command :: caused by :: operation exceeded time limit",
    "errName" : "MaxTimeMSExpired",
    "errCode" : 50.0,
    "responseLength" : 0.0,
    "protocol" : "op_msg",
    "millis" : 1002.0,
    "planSummary" : "COLLSCAN",
    "ts" : ISODate("2024-07-09T03:45:55.304+0000"),
    "client" : "0.0.0.0",
    "allUsers" : [
        {
            "user" : "__system",
            "db" : "local"
        }
    ],
    "user" : "__system@local"
}

Why would MongoDB initiate such queries on a secondary server?