Removing a Time Series collection killed two thirds of a shard

Okay, so I’m using Mongo 6.X on RHEL 7 (AWS EC2 (on a really “beefy” instance)). I created a sharded time-series collection and named it with a _2 at the end of the name (which should be okay). I went to delete it (and it had zero rows), and the command hung for several minutes, so I pulled open another terminal and my mongod executables on 2 ec2 instances had just died. This was in their logfile:

{"t":{"$date":"2024-02-20T03:53:22.432Z"},"s":"F",  "c":"REPL",     "id":21238,   "ctx":"ReplWriterWorker-187","msg":"Writer worker caught exception","attr":{"error":"CommandNotSupportedOnView: Namespace someobservation.someobservation_2 is a timeseries collection","oplogEntry":{"oplogEntry":{"op":"d","ns":"config.cache.collections","ui":{"$uuid":"da4ebde6-04be-4c55-ba43-2a6d1bfba3cf"},"o":{"_id":"someobservation.someobservation_2"},"ts":{"$timestamp":{"t":1708401202,"i":15}},"t":50,"v":2,"wall":{"$date":"2024-02-20T03:53:22.430Z"}}}}}
{"t":{"$date":"2024-02-20T03:53:22.432Z"},"s":"F",  "c":"REPL",     "id":21235,   "ctx":"OplogApplier-0","msg":"Failed to apply batch of operations","attr":{"numOperationsInBatch":1,"firstOperation":{"oplogEntry":{"op":"d","ns":"config.cache.collections","ui":{"$uuid":"da4ebde6-04be-4c55-ba43-2a6d1bfba3cf"},"o":{"_id":"someobservation.someobservation_2"},"ts":{"$timestamp":{"t":1708401202,"i":15}},"t":50,"v":2,"wall":{"$date":"2024-02-20T03:53:22.430Z"}}},"lastOperation":{"oplogEntry":{"op":"d","ns":"config.cache.collections","ui":{"$uuid":"da4ebde6-04be-4c55-ba43-2a6d1bfba3cf"},"o":{"_id":"someobservation.someobservation_2"},"ts":{"$timestamp":{"t":1708401202,"i":15}},"t":50,"v":2,"wall":{"$date":"2024-02-20T03:53:22.430Z"}}},"failedWriterThread":3,"error":"CommandNotSupportedOnView: Namespace someobservation.someobservation_2 is a timeseries collection"}}
{"t":{"$date":"2024-02-20T03:53:22.432Z"},"s":"F",  "c":"ASSERT",   "id":23095,   "ctx":"OplogApplier-0","msg":"Fatal assertion","attr":{"msgid":34437,"error":"CommandNotSupportedOnView: Namespace someobservation.someobservation_2 is a timeseries collection","file":"src/mongo/db/repl/oplog_applier_impl.cpp","line":345}}
{"t":{"$date":"2024-02-20T03:53:22.432Z"},"s":"F",  "c":"ASSERT",   "id":23096,   "ctx":"OplogApplier-0","msg":"\n\n***aborting after fassert() failure\n\n"}

When I tried to restart, it would just throw the same error (mainly some C++ looking errors) and failed to restart. I then disabled the mongod process, did a “rm-rf *” in the data directory, restarted the process and it was re-populated and worked (and the time-series collection was removed). Anyway, I just thought I’d put this out there just in case someone else encountered this same issue.

What command did you run to delete?

Well, in NoSQLBooster, I just right clicked on the collection then selected to drop it. NoSQLBooster does a particularly good job about using the correct syntax (in this case db.collection.drop().