Hey, I’ve been thinking about using mongo for my project to logging everything but I couldn’t decide that I should really us mongo for logging my data or not. Therefore, I want to ask here and get some experienced people advices.
Firstly, I want to log everything that happens during working. Thus, there’d be a lot of logs, there’d be hundreds of thousands log data. Logs will not be that large in the memory(few fields like date, user etc.)
Should I use only one collection to log everything or separate every log by their category?
Should I use expired data? (If it’s possible, I’d like to see logs which are have stayed longer than a year too.)
My application only write logs to mongo. I don’t get/update/delete logs right now.
I’ve look documents you attached. I really want to know which way is more effective for my project. Basically, I log everything about my project and I want to know the question I asked above.
Especially, I’d like to know how to store log for a long time(10 years.) without performance impact.
In that case I think you can write the logs as single documents.
Possibly partition the data into weekly or monthly collections.
Drop collections once they past 10 years.
What are the volume of the events you log? How many per minute?
Please note that MongoDB Atlas has a datalake archiving features allowing you to archive collections to S3 Amazon buckets for cheap storage of large volumes.