Hi experts,
the following errors keep appearing in our logs:
{"t":{"$date":"2021-11-12T11:37:01.263+00:00"},"s":"E", "c":"INDEX", "id":5400703, "ctx":"TTLMonitor","msg":"Error running TTL job on collection","attr":{"namespace":"config.system.sessions","error":{"code":301,"codeName":"DataCorruptionDetected","errmsg":"Erroneous index key found with reference to non-existent record id. Consider dropping and then re-creating the index and then running the validate command on the collection."}}}
{"t":{"$date":"2021-11-12T11:37:01.263+00:00"},"s":"E", "c":"QUERY", "id":4615603, "ctx":"TTLMonitor","msg":"Erroneous index key found with reference to non-existent record id. Consider dropping and then re-creating the index and then running the validate command on the collection.","attr":{"namespace":"config.system.sessions","recordId":"5465700","indexKeyData":[{"key":{"lastUse":{"$date":"2021-10-27T08:23:04.495Z"}},"pattern":{"lastUse":1}}]}}
{"t":{"$date":"2021-11-12T11:37:01.051+00:00"},"s":"I", "c":"COMMAND", "id":51803, "ctx":"LogicalSessionCacheRefresh","msg":"Slow query","attr":{"type":"command","ns":"config.$cmd","command":{"update":"system.sessions","ordered":false,"writeConcern":{"w":"majority","wtimeout":15000},"$db":"config"},"numYields":0,"reslen":3160,"locks":{"ParallelBatchWriterMode":{"acquireCount":{"r":33}},"ReplicationStateTransition":{"acquireCount":{"w":35}},"Global":{"acquireCount":{"r":3,"w":33}},"Database":{"acquireCount":{"r":2,"w":33}},"Collection":{"acquireCount":{"r":2,"w":33}},"Mutex":{"acquireCount":{"r":34}}},"flowControl":{"acquireCount":33,"timeAcquiringMicros":38},"writeConcern":{"w":"majority","wtimeout":15000,"provenance":"clientSupplied"},"storage":{"data":{"bytesRead":217956,"timeReadingMicros":96293}},"protocol":"op_msg","durationMillis":107}}
The error message basically says that the index contains an outdated entry, i.e. a key pointing to a record that no longer exists. It also suggest to drop and re-create the index.
Fine, but the documentation https://docs.mongodb.com/manual/reference/config-database/#collections-to-support-sessions explicitly states NOT to drop to config.system.sessions.
So what is the correct solution here?
And how can we avoid having this “data corruption” again?
Any thoughts are much appreciated