While performing a "Mongorestore", the operation fails at 35% and throws an error of "Socket say send() errno:104 Connection reset by peer 127.0.0.1:27017 assertion: 9001 socket exception [SEND_ERROR] server [127.0.0.1:27017]"

Objective: To restore the mongodump of 3.81 TB into Mongodb 2.6.12 hosted on an Azure Ubuntu 20.04 server with a RAM of 16 GB. The Open files limits have been set to 70 Million and other limits have been maxed-out.

After getting this error, even the Mongo commands doesn’t work inside the Mongod shell and says:

2023-08-07T04:24:00.137+0000 DBClientCursor::init call() failed
2023-08-07T04:24:00.138+0000 Error: error doing query: failed at src/mongo/shell/query.js:81
2023-08-07T04:24:00.139+0000 trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2023-08-07T04:24:00.140+0000 reconnect 127.0.0.1:27017 (127.0.0.1) ok
2023-08-07T04:24:00.140+0000 Socket recv() errno:104 Connection reset by peer 127.0.0.1:27017
2023-08-07T04:24:00.140+0000 SocketException: remote: 127.0.0.1:27017 error: 9001 socket exception [RECV_ERROR] server [127.0.0.1:27017]
2023-08-07T04:24:00.140+0000 DBClientCursor::init call() failed

I tried to restart the Mongod service inorder to fix these connection reset issues, but then the mongorestore again fails at same point. After checking the logs it says something different:

2023-08-07T00:43:28.872+0000 [journal] flushing directory /var/lib/mongodb/journal
2023-08-07T00:43:28.874+0000 [journal] Assertion: 13650:Couldn't open directory '/var/lib/mongodb/journal' for flushing: errno:24 Too many open files
2023-08-07T00:43:28.894+0000 [journal] 0x121df81 0x11bd689 0x11a23e6 0x11a294c 0x121639a 0x11be183 0xa73d4b 0xa7764a 0xa779c2 0xa77c22 0xa69f58 0xa6c609 0xa6c947 0x1262d49 0x7f10ee1ab609 0x7f10edd7a133 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x121df81]
 /usr/bin/mongod(_ZN5mongo10logContextEPKc+0x159) [0x11bd689]
 /usr/bin/mongod(_ZN5mongo11msgassertedEiPKc+0xe6) [0x11a23e6]
 /usr/bin/mongod() [0x11a294c]
 /usr/bin/mongod(_ZN5mongo16flushMyDirectoryERKN5boost11filesystem34pathE+0x73a) [0x121639a]
 /usr/bin/mongod(_ZN5mongo7LogFileC1ERKSsb+0x83) [0x11be183]
 /usr/bin/mongod(_ZN5mongo3dur7Journal5_openEv+0x1db) [0xa73d4b]
 /usr/bin/mongod(_ZN5mongo3dur7Journal7_rotateEy+0x8a) [0xa7764a]
 /usr/bin/mongod(_ZN5mongo3dur7Journal7journalERKNS0_11JSectHeaderERKNS_14AlignedBuilderE+0x1f2) [0xa779c2]
 /usr/bin/mongod(_ZN5mongo3dur14WRITETOJOURNALENS0_11JSectHeaderERNS_14AlignedBuilderE+0x32) [0xa77c22]
 /usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x158) [0xa69f58]
 /usr/bin/mongod() [0xa6c609]
 /usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x297) [0xa6c947]
 /usr/bin/mongod() [0x1262d49]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f10ee1ab609]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f10edd7a133]
2023-08-07T00:43:28.897+0000 [journal] warning couldn't write to / rename file /var/lib/mongodb/journal/prealloc.1: Couldn't open directory '/var/lib/mongodb/journal' for flushing: errno:24 Too many open files
2023-08-07T00:43:28.897+0000 [journal] User Assertion: 13516:couldn't open file /var/lib/mongodb/journal/j._1381 for writing errno:24 Too many open files
2023-08-07T00:43:28.897+0000 [journal] warning exception opening journal file couldn't open file /var/lib/mongodb/journal/j._1381 for writing errno:24 Too many open files
2023-08-07T00:43:28.897+0000 [journal] error exception in dur::journal couldn't open file /var/lib/mongodb/journal/j._1381 for writing errno:24 Too many open files
2023-08-07T00:43:28.897+0000 [journal] dbexception in groupCommitLL causing immediate shutdown: 13516 couldn't open file /var/lib/mongodb/journal/j._1381 for writing errno:24 Too many open files
2023-08-07T00:43:28.897+0000 [journal] SEVERE: dur1
2023-08-07T00:43:28.897+0000 [conn2] insert ps.Images ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:587 461ms
2023-08-07T00:43:28.898+0000 [conn2] insert ps.Images ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:611 0ms
2023-08-07T00:43:28.898+0000 [conn2] insert ps.Images ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:306 0ms
2023-08-07T00:43:28.899+0000 [conn2] insert ps.Images ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:419 0ms
2023-08-07T00:43:28.900+0000 [conn2] insert ps.Images ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:520 0ms
2023-08-07T00:43:28.900+0000 [conn2] insert ps.Images ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:535 0ms
2023-08-07T00:43:28.901+0000 [conn2] insert ps.Images ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:717 0ms
2023-08-07T00:43:28.902+0000 [conn2] insert ps.Images ninserted:1 keyUpdates:0 numYields:0 locks(micros) w:260 0ms
2023-08-07T00:43:28.902+0000 [journal] SEVERE: Got signal: 6 (Aborted).
Backtrace:0x121df81 0x121d35e 0x7f10edc9e090 0x7f10edc9e00b 0x7f10edc7d859 0xba87eb 0xa6a3e6 0xa6c609 0xa6c947 0x1262d49 0x7f10ee1ab609 0x7f10edd7a133 
 /usr/bin/mongod(_ZN5mongo15printStackTraceERSo+0x21) [0x121df81]
 /usr/bin/mongod() [0x121d35e]
 /lib/x86_64-linux-gnu/libc.so.6(+0x43090) [0x7f10edc9e090]
 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7f10edc9e00b]
 /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7f10edc7d859]
 /usr/bin/mongod(_ZN5mongo10mongoAbortEPKc+0x6b) [0xba87eb]
 /usr/bin/mongod(_ZN5mongo3dur27groupCommitWithLimitedLocksEv+0x5e6) [0xa6a3e6]
 /usr/bin/mongod() [0xa6c609]
 /usr/bin/mongod(_ZN5mongo3dur9durThreadEv+0x297) [0xa6c947]
 /usr/bin/mongod() [0x1262d49]
 /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f10ee1ab609]
 /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f10edd7a133]
2023-08-07T00:43:36.331+0000 ***** SERVER RESTARTED *****
2023-08-07T00:43:36.334+0000 [DataFileSync] BackgroundJob starting: DataFileSync
2023-08-07T00:43:36.335+0000 shardKeyTest passed
2023-08-07T00:43:36.336+0000 isInRangeTest passed
2023-08-07T00:43:36.336+0000 shardObjTest passed
2023-08-07T00:43:36.336+0000 [initandlisten] MongoDB starting : pid=187795 port=27017 dbpath=/var/lib/mongodb 64-bit host=retblt-q-vm-02
2023-08-07T00:43:36.336+0000 [initandlisten] db version v2.6.12
2023-08-07T00:43:36.336+0000 [initandlisten] git version: d73c92b1c85703828b55c2916a5dd4ad46535f6a
2023-08-07T00:43:36.336+0000 [initandlisten] build info: Linux build5.ny.cbi.10gen.cc 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 BOOST_LIB_VERSION=1_49
2023-08-07T00:43:36.336+0000 [initandlisten] allocator: tcmalloc
2023-08-07T00:43:36.336+0000 [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", http: { enabled: false }, port: 27017 }, processManagement: { pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", verbosity: 1 } }
2023-08-07T00:43:36.525+0000 [initandlisten] flushing directory /var/lib/mongodb
2023-08-07T00:43:36.876+0000 [initandlisten] journal dir=/var/lib/mongodb/journal
2023-08-07T00:43:36.877+0000 [initandlisten] recover begin
2023-08-07T00:43:36.921+0000 [initandlisten] recover lsn: 32985510
2023-08-07T00:43:36.921+0000 [initandlisten] recover /var/lib/mongodb/journal/j._1377
2023-08-07T00:43:40.133+0000 [initandlisten] recover skipping application of section seq:32962980 < lsn:32985510
2023-08-07T00:43:43.380+0000 [initandlisten] recover skipping application of section seq:32963320 < lsn:32985510
2023-08-07T00:43:46.479+0000 [initandlisten] recover skipping application of section seq:32963610 < lsn:32985510
2023-08-07T00:43:46.754+0000 [initandlisten] recover skipping application of section seq:32963960 < lsn:32985510
2023-08-07T00:43:46.986+0000 [initandlisten] recover skipping application of section seq:32964320 < lsn:32985510
2023-08-07T00:43:47.182+0000 [initandlisten] recover skipping application of section seq:32964700 < lsn:32985510
2023-08-07T00:43:47.414+0000 [initandlisten] recover skipping application of section seq:32965040 < lsn:32985510
2023-08-07T00:43:47.608+0000 [initandlisten] recover skipping application of section seq:32965390 < lsn:32985510
2023-08-07T00:43:47.899+0000 [initandlisten] recover skipping application of section seq:32965690 < lsn:32985510
2023-08-07T00:43:48.120+0000 [initandlisten] recover skipping application of section more...
2023-08-07T00:43:53.796+0000 [initandlisten] recover final skipped journal section had sequence number 32975340
2023-08-07T00:43:53.796+0000 [initandlisten] recover applying initial journal section with sequence number 32975630
2023-08-07T00:44:16.204+0000 [initandlisten] recover create file /var/lib/mongodb/ps.997 2047MB
2023-08-07T00:44:28.186+0000 [initandlisten] flushing directory /var/lib/mongodb
2023-08-07T00:44:28.608+0000 [initandlisten] recover /var/lib/mongodb/journal/j._1378
2023-08-07T00:44:55.558+0000 [initandlisten] recover /var/lib/mongodb/journal/j._1379
2023-08-07T00:45:06.127+0000 [initandlisten] recover create file /var/lib/mongodb/ps.998 2047MB
2023-08-07T00:45:27.492+0000 [initandlisten] flushing directory /var/lib/mongodb
2023-08-07T00:45:36.785+0000 [initandlisten] recover /var/lib/mongodb/journal/j._1380
2023-08-07T00:46:05.695+0000 [initandlisten] recover create file /var/lib/mongodb/ps.999 2047MB
2023-08-07T00:46:27.439+0000 [initandlisten] flushing directory /var/lib/mongodb
2023-08-07T00:46:32.192+0000 [initandlisten] recover cleaning up
2023-08-07T00:46:32.192+0000 [initandlisten] removeJournalFiles
2023-08-07T00:46:32.587+0000 [initandlisten] flushing directory /var/lib/mongodb/journal
2023-08-07T00:46:32.592+0000 [initandlisten] removeJournalFiles end
2023-08-07T00:46:32.592+0000 [initandlisten] recover done
2023-08-07T00:46:32.592+0000 [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.0
2023-08-07T00:46:32.592+0000 [DataFileSync] flushing mmaps took 116257ms  for 0 files
2023-08-07T00:46:32.592+0000 [DataFileSync] flushing mmaps took 0ms  for 0 files
2023-08-07T00:46:43.144+0000 [initandlisten] preallocating a journal file /var/lib/mongodb/journal/prealloc.2
2023-08-07T00:46:54.135+0000 [initandlisten] flushing directory /var/lib/mongodb/journal
2023-08-07T00:46:54.136+0000 [initandlisten] flushing directory /var/lib/mongodb/journal
2023-08-07T00:46:54.143+0000 [initandlisten] opening db:  local
2023-08-07T00:46:54.181+0000 [initandlisten] enter repairDatabases (to check pdfile version #)
2023-08-07T00:46:54.182+0000 [initandlisten] 	ps
2023-08-07T00:46:54.182+0000 [initandlisten] opening db:  ps
2023-08-07T00:46:54.352+0000 [initandlisten] 	ps_digital_postage
2023-08-07T00:46:54.352+0000 [initandlisten] opening db:  ps_digital_postage
2023-08-07T00:46:54.410+0000 [initandlisten] 	local
2023-08-07T00:46:54.432+0000 [initandlisten] 	admin
2023-08-07T00:46:54.432+0000 [initandlisten] opening db:  admin
2023-08-07T00:46:54.470+0000 [initandlisten] done repairDatabases
2023-08-07T00:46:54.470+0000 [initandlisten] opening db:  admin
2023-08-07T00:46:54.487+0000 [initandlisten] query admin.system.roles planSummary: COLLSCAN ntoreturn:0 ntoskip:0 nscanned:0 nscannedObjects:0 keyUpdates:0 numYields:0 locks(micros) W:1410 r:15162 nreturned:0 reslen:20 15ms
2023-08-07T00:46:54.487+0000 [TTLMonitor] BackgroundJob starting: TTLMonitor
2023-08-07T00:46:54.487+0000 [PeriodicTaskRunner] BackgroundJob starting: PeriodicTaskRunner
2023-08-07T00:46:54.487+0000 [ClientCursorMonitor] BackgroundJob starting: ClientCursorMonitor
2023-08-07T00:46:54.503+0000 [initandlisten] fd limit hard:524288 soft:1024 max conn: 819
2023-08-07T00:46:54.503+0000 [IndexRebuilder] BackgroundJob starting: IndexRebuilder
2023-08-07T00:46:54.505+0000 [initandlisten] opening db:  local
2023-08-07T00:46:54.506+0000 [initandlisten] create collection local.startup_log { size: 10485760, capped: true }
2023-08-07T00:46:54.511+0000 [initandlisten] command local.$cmd command: create { create: "startup_log", size: 10485760, capped: true } ntoreturn:1 keyUpdates:0 numYields:0  reslen:75 6ms
2023-08-07T00:46:54.511+0000 [initandlisten] insert local.startup_log ninserted:1 keyUpdates:0 numYields:0  0ms
2023-08-07T00:46:54.511+0000 [IndexRebuilder] opening db:  ps
2023-08-07T00:46:54.511+0000 [initandlisten] waiting for connections on port 27017
2023-08-07T00:46:54.574+0000 [journal] lsn set 0
2023-08-07T00:46:54.737+0000 [IndexRebuilder] opening db:  ps_digital_postage
2023-08-07T00:46:55.285+0000 [IndexRebuilder] checking complete
2023-08-07T00:47:32.598+0000 [DataFileSync] flushing mmaps took 6ms  for 1012 files

Can someone please suggest how to resolve the above error ?