Mongo Crashes due to "Too many open files" error

Ops and Admin
1

Hello everyone,

I started facing an issue recently. My company gave a computer (Macbook Pro 2019 Catalina 10.15.7). I set everything up as my personal environment. The only difference I haven’t format my personal environment for a long time, and this one recently bought and set up. Therefore the all project environment and settings for mongo is exactly same. After successful installation and run, after a while - which depends on the action I take on the projects - mongo crashes. Mongo logs are like following;

{"t":{"$date":"2021-02-22T06:56:42.725+03:00"},"s":"I",  "c":"NETWORK",  "id":22944,   "ctx":"conn112","msg":"Connection ended","attr":{"remote":"127.0.0.1:52506","connectionId":112,"connectionCount":108}}
{"t":{"$date":"2021-02-22T06:56:51.443+03:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"127.0.0.1:52517","connectionId":113,"connectionCount":109}}
{"t":{"$date":"2021-02-22T06:56:51.444+03:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn113","msg":"client metadata","attr":{"remote":"127.0.0.1:52517","client":"conn113","doc":{"driver":{"name":"nodejs","version":"3.6.3"},"os":{"type":"Darwin","name":"darwin","architecture":"x64","version":"19.6.0"},"platform":"'Node.js v10.18.0, LE (unified)"}}}
{"t":{"$date":"2021-02-22T06:56:52.219+03:00"},"s":"I",  "c":"NETWORK",  "id":23018,   "ctx":"listener","msg":"Error accepting new connection on local endpoint","attr":{"localEndpoint":"127.0.0.1:27017","error":"Too many open files"}}
{"t":{"$date":"2021-02-22T06:56:52.696+03:00"},"s":"I",  "c":"NETWORK",  "id":23018,   "ctx":"listener","msg":"Error accepting new connection on local endpoint","attr":{"localEndpoint":"127.0.0.1:27017","error":"Too many open files"}}
{"t":{"$date":"2021-02-22T06:56:52.730+03:00"},"s":"I",  "c":"NETWORK",  "id":23018,   "ctx":"listener","msg":"Error accepting new connection on local endpoint","attr":{"localEndpoint":"127.0.0.1:27017","error":"Too many open files"}}
{"t":{"$date":"2021-02-22T06:56:52.962+03:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"thread114","msg":"WiredTiger error","attr":{"error":24,"message":"[1613966212:962628][7406:0x70000bb45000], log-server: __directory_list_worker, 46: /usr/local/var/mongodb/journal: directory-list: opendir: Too many open files"}}
{"t":{"$date":"2021-02-22T06:56:52.962+03:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"thread114","msg":"WiredTiger error","attr":{"error":24,"message":"[1613966212:962756][7406:0x70000bb45000], log-server: __log_prealloc_once, 505: log pre-alloc server error: Too many open files"}}
{"t":{"$date":"2021-02-22T06:56:52.962+03:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"thread114","msg":"WiredTiger error","attr":{"error":24,"message":"[1613966212:962784][7406:0x70000bb45000], log-server: __log_server, 961: log server error: Too many open files"}}
{"t":{"$date":"2021-02-22T06:56:52.962+03:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"thread114","msg":"WiredTiger error","attr":{"error":-31804,"message":"[1613966212:962806][7406:0x70000bb45000], log-server: __log_server, 961: the process must exit and restart: WT_PANIC: WiredTiger library panic"}}
{"t":{"$date":"2021-02-22T06:56:52.963+03:00"},"s":"F",  "c":"-",        "id":23089,   "ctx":"thread114","msg":"Fatal assertion","attr":{"msgid":50853,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp","line":520}}
{"t":{"$date":"2021-02-22T06:56:52.963+03:00"},"s":"F",  "c":"-",        "id":23090,   "ctx":"thread114","msg":"\n\n***aborting after fassert() failure\n\n"}
{"t":{"$date":"2021-02-22T06:56:52.964+03:00"},"s":"F",  "c":"CONTROL",  "id":4757800, "ctx":"thread114","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Abort trap: 6).\n"}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31431,   "ctx":"thread114","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[{"a":"105A35B9C","b":"1038CA000","o":"216BB9C","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE","s+":"10C"},{"a":"105A372A8","b":"1038CA000","o":"216D2A8","s":"_ZN5mongo15printStackTraceEv","s+":"28"},{"a":"105A34DDB","b":"1038CA000","o":"216ADDB","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9__siginfoPv","s+":"BB"},{"a":"7FFF73CF75FD","b":"7FFF73CF4000","o":"35FD","s":"_sigtramp","s+":"1D"},{"a":"0"},{"a":"7FFF73BCD808","b":"7FFF73B4E000","o":"7F808","s":"abort","s+":"78"},{"a":"105A1C2D7","b":"1038CA000","o":"21522D7","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"197"},{"a":"1039976FB","b":"1038CA000","o":"CD6FB","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc","s+":"1FB"},{"a":"103B02AE7","b":"1038CA000","o":"238AE7","s":"__eventv","s+":"607"},{"a":"103B02D66","b":"1038CA000","o":"238D66","s":"__wt_panic_func","s+":"FD"},{"a":"103A1E59E","b":"1038CA000","o":"15459E","s":"__log_server","s+":"44E"},{"a":"7FFF73D03109","b":"7FFF73CFD000","o":"6109","s":"_pthread_start","s+":"94"},{"a":"7FFF73CFEB8B","b":"7FFF73CFD000","o":"1B8B","s":"thread_start","s+":"F"}],"processInfo":{"mongodbVersion":"4.4.3","gitVersion":"913d6b62acfbb344dde1b116f4161360acd8fd13","compiledModules":[],"uname":{"sysname":"Darwin","release":"19.6.0","version":"Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64","machine":"x86_64"},"somap":[{"path":"/usr/local/opt/mongodb-community/bin/mongod","machType":2,"b":"1038CA000","vmaddr":"100000000","buildId":"88F05A2CDBD83B9F98DAF635FC65C2E6"},{"path":"/usr/lib/system/libsystem_c.dylib","machType":6,"b":"7FFF73B4E000","vmaddr":"7FFF67253000","buildId":"BBDED5E6A6463EEDB33A91E4331EA063"},{"path":"/usr/lib/system/libsystem_platform.dylib","machType":6,"b":"7FFF73CF4000","vmaddr":"7FFF673F9000","buildId":"009A7C1F313A318EB9F230F4C06FEA5C"},{"path":"/usr/lib/system/libsystem_pthread.dylib","machType":6,"b":"7FFF73CFD000","vmaddr":"7FFF67402000","buildId":"62CB1A980B8F31E7A02BA1139927F61D"}]}}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"105A35B9C","b":"1038CA000","o":"216BB9C","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE","s+":"10C"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"105A372A8","b":"1038CA000","o":"216D2A8","s":"_ZN5mongo15printStackTraceEv","s+":"28"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"105A34DDB","b":"1038CA000","o":"216ADDB","s":"_ZN5mongo12_GLOBAL__N_116abruptQuitActionEiP9__siginfoPv","s+":"BB"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FFF73CF75FD","b":"7FFF73CF4000","o":"35FD","s":"_sigtramp","s+":"1D"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"0"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FFF73BCD808","b":"7FFF73B4E000","o":"7F808","s":"abort","s+":"78"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"105A1C2D7","b":"1038CA000","o":"21522D7","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj","s+":"197"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"1039976FB","b":"1038CA000","o":"CD6FB","s":"_ZN5mongo12_GLOBAL__N_141mdb_handle_error_with_startup_suppressionEP18__wt_event_handlerP12__wt_sessioniPKc","s+":"1FB"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"103B02AE7","b":"1038CA000","o":"238AE7","s":"__eventv","s+":"607"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"103B02D66","b":"1038CA000","o":"238D66","s":"__wt_panic_func","s+":"FD"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"103A1E59E","b":"1038CA000","o":"15459E","s":"__log_server","s+":"44E"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FFF73D03109","b":"7FFF73CFD000","o":"6109","s":"_pthread_start","s+":"94"}}}
{"t":{"$date":"2021-02-22T06:56:52.972+03:00"},"s":"I",  "c":"CONTROL",  "id":31427,   "ctx":"thread114","msg":"  Frame: {frame}","attr":{"frame":{"a":"7FFF73CFEB8B","b":"7FFF73CFD000","o":"1B8B","s":"thread_start","s+":"F"}}}

Here is node

MongoNetworkError: connection 1 to 127.0.0.1:27017 closed
    at (anonymous function).forEach.op (/Users/project/node_modules/mongodb/lib/cmap/connection.js:68:15)
    at Map.forEach (<anonymous>)
    at Socket.Connection.stream.on (/Users/project/node_modules/mongodb/lib/cmap/connection.js:67:20)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at TCP._handle.close (net.js:607:12) name: 'MongoNetworkError'

Limits are; ( it is exaclty same as my personal environment)

	cpu         unlimited      unlimited      
	filesize    unlimited      unlimited      
	data        unlimited      unlimited      
	stack       8388608        67104768       
	core        0              unlimited      
	rss         unlimited      unlimited      
	memlock     unlimited      unlimited      
	maxproc     2784           4176           
	maxfiles    256            unlimited
2

Hello @bardanadam, welcome to the community!

in your case the ulimit setting might be the issue. Please follow this link in the middle of the page you can find detailed information:

macOS

For macOS systems that have installed MongoDB Community using the brew installation method, the recommended open files value is automatically set when you start MongoDB through brew services . See Run MongoDB with brew for more information.

For macOS systems running MongoDb Enterprise or using the TGZ installation method, use the launchctl limit command to set the recommended values. See your operating system documentation for the precise procedure for changing system limits on running systems.

You can check your setting with unlimit -a recommended are:

  • 64000 for open files (option -n) and
  • 64000 for processes (option -u)

But before setting this manually please try to use the recommended startup options (see above, link)

Regards,
Michael

3

Hey @michael_hoeller,

Thank you for your help. However, I changed the default ulimit settings to the suggested value, which is 64000; since it’s e macOS, it allowed me to assign only 2500. After I set it up, I started still receiving the same type of error in a different message that says, “Connection refused because there are too many open connections.”

{"t":{"$date":"2021-02-23T16:06:41.382+03:00"},"s":"I",  "c":"NETWORK",  "id":22942,   "ctx":"listener","msg":"Connection refused because there are too many open connections","attr":{"connectionCount":820}}
{"t":{"$date":"2021-02-23T16:06:41.385+03:00"},"s":"I",  "c":"NETWORK",  "id":22942,   "ctx":"listener","msg":"Connection refused because there are too many open connections","attr":{"connectionCount":820}}
4

Hello @bardanadam
2500 feels very low, I have no Mac Laptop to verify but is does not sound ok. Are there any error messages? Access rights? Did you tried to start via brew service or rsp. used launchctl limit ? If not I recommend to do so. Sorry not much of help due to the fact that I have no Mac around.
Michael

5

Hey @michael_hoeller,

Yes, I can confirm the issue was ulimit settings. Also, I did what you suggested, and it works. The weird point is that my previous MacBook was working without any additional settings. I checked the limits and they were default. Anyway, thank you for your helping. I also wanted you to know the solution for the Mac too.

Best,
Burak

1 Like
6

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.