Update and Resolved:
Application Connection to MongoDB had one usecase that was opening files and connections and not closing them or using the connection pool that existed - Max open files were reached, all nodes went down, there was downtime and I had to increase the ulimit from 64k to 100k for max open files so that MongoDB could run for a bit until - restarted the usecase to remove all kept connections and pausing it until the developers take a look at it.