I am currently working on logging real-time data to a database with a frequency of 50 milliseconds. I need guidance on the appropriate approach for logging the data, considering the following scenarios:
- Each streaming result will vary, and I want to associate some metadata with each result.
- Throughout the day, I may have 12 streaming sessions per hour. However, the data collected within each hour will differ from data collected in other hours, as each hour will have a distinct name and additional metadata pertaining to the following hour’s streaming data.
Here is the approach I have tried so far:
- Initially, I created a separate collection for each log, with each streaming record stored as a document. However, this resulted in a large number of collections accumulating in the database over time.
- To address this issue, I attempted a bucket pattern. I stored the metadata of hourly streaming results in a document within a collection(So, for multiple hours I will have multiple metadata documents which are being referenced into another collection to store its’s streaming results), while the corresponding data was added to another collection using DBRef. However, I encountered a problem due to the maximum document size limit of 16 MB. To work around this, I checked the size of each document before inserting data into it. If the size reached 16 MB, I created a new document and added the streaming results to it. Unfortunately, this approach led to high CPU utilization, reaching up to 94%, and caused system freezes every hour.
Therefore, I am seeking potential solutions that can handle the 16 MB document size limitation while also addressing the CPU utilization issue.
I also have some specific questions regarding potential solutions:
- GridFS: Can I utilize GridFS in this case? Although I am not dealing with file data, I am streaming data through a socket. Would GridFS be applicable here?
- Time series databases: Will a new time series collection be created for each hour? If so, are there any limitations on the number of collections that can be created? Will this approach resolve the 16 MB document size issue? Additionally, what would be the suggested structure for time series documents? My requirement is to store metadata in one document and all streaming results in another document. (Does not matter whether I am storing in one collection or referenced collections- I want logical solution to it) Furthermore, I would like the ability to update the metadata later on. Is this feasible with a time series database?