Mongos service crashed

Hi All,

Today, In our production cluster, we faced an issue with MongoDB and couldn’t identify the issue. Kindly help me to identify the issue.

Issue Details:-
Mongos Service got interrupted almost the same time in all the nodes (3 node cluster).
Error Message

2020-06-30T23:48:16.413+0000 I  -        [conn1086250] operation was interrupted because a client disconnected
2020-06-30T23:48:16.443+0000 F  -        [conn1086250] terminate() called. No exception is active 0x55aa2cd50fe1 0x55aa2cd50d98 0x55aa2ce5b586 0x55aa2ce5b5c1 0x55aa2bfb557a 0x55aa2c238cd9 0x55aa2c239252 0x55aa2c159e00 0x55aa2c17df1c 0x55aa2c17806f 0x55aa2c17b2fc 0x55aa2c4f7c22 0x55aa2c175a6d 0x55aa2c178d23 0x55aa2c177137 0x55aa2c177fcb 0x55aa2c17b2fc 0x55aa2c4f808b 0x55aa2cbf3824 0x7f4333cafe65 0x7f43339d888d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55AA2BA30000","o":"1320FE1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55AA2BA30000","o":"1320D98"},{"b":"55AA2BA30000","o":"142B586","s":"_ZN10__cxxabiv111__terminateEPFvvE"},{"b":"55AA2BA30000","o":"142B5C1"},{"b":"55AA2BA30000","o":"58557A"},{"b":"55AA2BA30000","o":"808CD9"},{"b":"55AA2BA30000","o":"809252","s":"_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE"},{"b":"55AA2BA30000","o":"729E00","s":"_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"55AA2BA30000","o":"74DF1C","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"55AA2BA30000","o":"74806F","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55AA2BA30000","o":"74B2FC"},{"b":"55AA2BA30000","o":"AC7C22","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE"},{"b":"55AA2BA30000","o":"745A6D","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE"},{"b":"55AA2BA30000","o":"748D23","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"55AA2BA30000","o":"747137","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"55AA2BA30000","o":"747FCB","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"55AA2BA30000","o":"74B2FC"},{"b":"55AA2BA30000","o":"AC808B"},{"b":"55AA2BA30000","o":"11C3824"},{"b":"7F4333CA8000","o":"7E65"},{"b":"7F43338DA000","o":"FE88D","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.2.2", "gitVersion" : "a0bbbff6ada159e19298d37946ac8dc4b497eadf", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-1062.9.1.el7.x86_64", "version" : "#1 SMP Fri Dec 6 15:49:49 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "55AA2BA30000", "elfType" : 3, "buildId" : "A58FCE757C520F50A4A1BBBDE7E676CB6C5C160E" }, { "b" : "7FFE1D468000", "elfType" : 3, "buildId" : "B5A5458535A1397FA6BAAF5E8C13A6395426A1B2" }, { "b" : "7F43350D6000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "89C83CEB5DE5FDEC4F3DFBA4FCAACB53D747A998" }, { "b" : "7F4334EBD000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "3009B26B33156EAAF99787AA3DA0C6AE99649755" }, { "b" : "7F4334A5A000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "4CF1939F660008CFA869D8364651F31AACD2C1C4" }, { "b" : "7F43347E8000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "3B305C3BA17FE394862E749763F2956C9C890C2E" }, { "b" : "7F43345E4000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "18113E6E83D8E981B8E8D808F7F3DBB23F950A1D" }, { "b" : "7F43343DC000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "4749697BF078337576C4629F0D30B296A0939779" }, { "b" : "7F43340DA000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5681C054FDABCF789F4DDA66E94F1F6ED1747327" }, { "b" : "7F4333EC4000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "DAC0179F4555AEFEC9E97476201802FD20C03EC5" }, { "b" : "7F4333CA8000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "8B33F7F8C86F8D544C63C5541A8E42B3DDFEF8B1" }, { "b" : "7F43338DA000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "398944D32CF16A67AF51067A326E6C0CC14F90ED" }, { "b" : "7F4335340000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5CC1A53B747A7E4D21198723C2B633E54F3C06D9" }, { "b" : "7F43336A7000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7F433347A000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "1AF123CADB2F2910E89CBD540A06D3B33692F95E" }, { "b" : "7F4333221000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "B6321C434B5C7386B144B925CEE2798D269FDDF5" }, { "b" : "7F4332FF9000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "BDA454441F59F41D2DA36E13CEA1FC4CE95B2BBB" }, { "b" : "7F4332CCA000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "DC3B36B530F506DE4FC1A6612D7DF44D4A3DDCDB" }, { "b" : "7F4332A9A000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "32C8FB6C2768FFE41E0A15CBF2089A4202CA2290" }, { "b" : "7F4332896000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "325B8CE57A776DE0B24B362A7E0C90E903B1A4B8" }, { "b" : "7F4332691000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "0460FF10A3C63749113D380C40E10DFCF066C76E" }, { "b" : "7F4332453000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "8840B019EDB66B0CFBD2F77EF196440F7928106E" }, { "b" : "7F4332206000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "E2AA8CA3D3164E7DBEC293BFA0B55D2B10DAC05D" }, { "b" : "7F4331F1D000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "3EE7267AF7BFD3B132E6A222D997DA09C96C90DD" }, { "b" : "7F4331CEA000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "82E28CACB60C27CD6F14A6D2268F0CFF621664D0" }, { "b" : "7F4331AE6000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "67E935BFABA2C914C01156B88947DD515EA51170" }, { "b" : "7F43318D7000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "3192C56CD451E18EB9F29CB045432BA9C738DD29" }, { "b" : "7F4331682000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "F1FADDDE0D21D5F4E2DCADEDD3B85B6E7AAC9883" }, { "b" : "7F433146C000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7F433125C000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "4F5FBB2087BE132892467C4E7A46A3D07E5DA40B" }, { "b" : "7F4331058000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F4330E3B000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E2F2017F821DD1B9D307DA1A9B8014F2941AEB7B" }, { "b" : "7F4330C14000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179" }, { "b" : "7F43309DD000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "84467C988F41D853C58353BEB247670E15DA8BAD" }, { "b" : "7F433077B000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7F4330578000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "197680DAE6538245CB99723E57447C4EF2E98362" }, { "b" : "7F4330365000", "path" : "/lib64/libnss_files.so.2", "elfType" : 3, "buildId" : "A1DB0E8103DE9F2540788EEA6CBCE3F639C2B39D" } ] }}
 mongos(_ZN5mongo15printStackTraceERSo+0x41) [0x55aa2cd50fe1]
 mongos(+0x1320D98) [0x55aa2cd50d98]
 mongos(_ZN10__cxxabiv111__terminateEPFvvE+0x6) [0x55aa2ce5b586]
 mongos(+0x142B5C1) [0x55aa2ce5b5c1]
 mongos(+0x58557A) [0x55aa2bfb557a]
 mongos(+0x808CD9) [0x55aa2c238cd9]
 mongos(_ZN5mongo8Strategy13clientCommandEPNS_16OperationContextERKNS_7MessageE+0x1C2) [0x55aa2c239252]
 mongos(_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE+0x3D0) [0x55aa2c159e00]
 mongos(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0xEC) [0x55aa2c17df1c]
 mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x17F) [0x55aa2c17806f]
 mongos(+0x74B2FC) [0x55aa2c17b2fc]
 mongos(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsENS0_23ServiceExecutorTaskNameE+0x182) [0x55aa2c4f7c22]
 mongos(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS2_23ServiceExecutorTaskNameENS0_9OwnershipE+0x10D) [0x55aa2c175a6d]
 mongos(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0x843) [0x55aa2c178d23]
 mongos(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x2E7) [0x55aa2c177137]
 mongos(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xDB) [0x55aa2c177fcb]
 mongos(+0x74B2FC) [0x55aa2c17b2fc]
 mongos(+0xAC808B) [0x55aa2c4f808b]
 mongos(+0x11C3824) [0x55aa2cbf3824]
 libpthread.so.0(+0x7E65) [0x7f4333cafe65]
 libc.so.6(clone+0x6D) [0x7f43339d888d]
-----  END BACKTRACE  -----

Please note that I couldn’t see any issue at Shard & Config Server Side at the same time.

Kindly let me know in case of any other details required.

Thanks.

Welcome to the community @Ann_Pricks_Edmund!

Please provide more details on your deployment:

  • MongoDB server version
  • O/S version

If you are running MongoDB 4.2.6 or an earlier version of 4.2.x, you are likely encountering SERVER-47553: mongos crashes due to client disconnecting when signing keys being refreshed. A fix for this issue is included in MongoDB 4.2.7 and newer.

Regards,
Stennie

1 Like

Thank You for your kind response.

Our MongoDB Version - MongoDB v4.2.2
O/S Version - CentOS Linux release 7.7.1908 (Core)

As per the reported bug, we are also encountering the same. We will plan for an upgrade.

Thanks.