BsonObjTooLarge error after restarting Mongod instance for Timeseries data

Hello Team,
We are currently running on Mongodb Community Version 7.0.2 With 3 replica Setup.
Each Replica Has 128 GB ram, 12 CPU and 2 TB SSD.
We have 120M socket data comes from devices and we are running on Timeseries collection per day.

I have started getting CachePressure and unnecessary bucket termination without filling buckets properly. NumberOfBucketsClosedDueToCachePressure metric was getting bigger in time.

So Then I have restarted Mongod Instance and It gives me some buffer to investigate.
I saw some errors on logs below,

{"t":{"$date":"2024-02-19T20:13:39.063+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5442","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 1634887031 (0x61726577) is invalid. Size must be between 0 and 16793600(16MB) First element: ","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:15:47.396+03:00"},"s":"I",  "c":"CONNPOOL", "id":22572,   "ctx":"MirrorMaestro","msg":"Dropping all pooled connections","attr":{"hostAndPort":"REDACTES","error":"ConnectionPoolExpired: Pool for REDACTEDhas expired."}}
{"t":{"$date":"2024-02-19T20:18:59.646+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5494","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 0 (0x0) is invalid. Size must be between 0 and 67108864(64MB) First element: EOO","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:18:59.646+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5494","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 1919250541 (0x7265706D) is invalid. Size must be between 0 and 67108864(64MB) First element: ","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:18:59.646+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5494","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 1701603681 (0x656C6961) is invalid. Size must be between 0 and 67108864(64MB) First element: ","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:18:59.646+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5494","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 1701603688 (0x656C6968) is invalid. Size must be between 0 and 16793600(16MB) First element: ","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:21:35.145+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5505","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 0 (0x0) is invalid. Size must be between 0 and 67108864(64MB) First element: EOO","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:21:35.145+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5505","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 1919252073 (0x72657669) is invalid. Size must be between 0 and 67108864(64MB) First element: ","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:21:35.145+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5505","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 1919252080 (0x72657670) is invalid. Size must be between 0 and 16793600(16MB) First element: ","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:26:01.440+03:00"},"s":"I",  "c":"CONNPOOL", "id":22572,   "ctx":"MirrorMaestro","msg":"Dropping all pooled connections","attr":{"hostAndPort":"REDACTED","error":"ConnectionPoolExpired: Pool for REDACTED has expired."}}
{"t":{"$date":"2024-02-19T20:28:55.403+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5644","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 0 (0x0) is invalid. Size must be between 0 and 67108864(64MB) First element: EOO","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:28:55.403+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5644","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 0 (0x0) is invalid. Size must be between 0 and 67108864(64MB) First element: ","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:28:55.403+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5644","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 1634887024 (0x61726570) is invalid. Size must be between 0 and 67108864(64MB) First element: ","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:28:55.403+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5644","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 1634887031 (0x61726577) is invalid. Size must be between 0 and 16793600(16MB) First element: ","file":"src/mongo/bson/bsonobj.cpp","line":104}}
{"t":{"$date":"2024-02-19T20:32:35.971+03:00"},"s":"I",  "c":"CONNPOOL", "id":22572,   "ctx":"MirrorMaestro","msg":"Dropping all pooled connections","attr":{"hostAndPort":"REDACTED","error":"ConnectionPoolExpired: Pool for REDACTED has expired."}}
{"t":{"$date":"2024-02-19T20:40:34.517+03:00"},"s":"E",  "c":"ASSERT",   "id":23077,   "ctx":"conn5789","msg":"Assertion","attr":{"error":"BSONObjectTooLarge: BSONObj size: 0 (0x0) is invalid. Size must be between 0 and 67108864(64MB) First element: 

So this errors occurs around every 15-20 min. But I’m not inserting any document over 16MB or anything. Servers are not actually getting any read queries yet. So I currenly only have write operations.

I’m suspecting that once I restart the instance, I have now more open buckets which are eligible to get the data. So when the data comes, It searches through more filled buckets and causing this issue. But I’m not Mongodb expert and I have no other clue how to investigate it further.

About the Cache Pressure, the details are here Please take a look if it seems related,

Any help would be appreciated.

Hello @Rachel_Vance
Thank you for your recommendations. I have played along with cache settings via wiredtiger, Changed some other settings that might caused this issue. But this issue is not related any generic utilization parameters on mongo. If you have specific setting you want me to tweak, I would happily do it.
My application doesn’t face this issue while inserting datas into mongo, It s some internal error. No idea what’s causing.

Thanks.