Storing ~20 Billion documents

Hi @Sason_Braha,

I would say that having 5000 nested documents in one array is also very high. It may impact many operations as MongoDB has to serelize and desirialize those arrays in many operations against the documents (cpu/ memory overhead).

Also it impose a high risk on expiration mechanism as constantly pulling and pushing array elements to large arrays is unadvisable…

I would consider keeping each document in a seperate one and indexing a field named uuid. If your HW will not be able to operate with this design consider sharding the environment on hash shard key for this uuid if you only have to query based on the uuid.

This way you can have a TTL index as your createDate will be on the main level of a document.

{
_id : ObjectID,
uuid: ... ,
createdDate: ...,
data: ...,
...
}

Best
Pavel