We are designing a new database (mongo or maybe mysql likely) for a log view project , i.e. ingesting large amount of logs (from cpu chips test logs) into the database and massage the data and make it more useful, couple of extra needs is the new table columns may need to add from time to time and the logs keep comings and need to parse and load. What primary design and tools and scrips to do you suggest? THX
Hi @Frank_Mostadam ,
This definitely sounds like a good fit for MongoDB timeseries collections. This collection type together with a bulkInsert API of the relevant driver will make data ingestion and analysis over time convenient while storage foot print is optimised and minimal.
MongoDB 5.2 now released on Atlas adds a columnar store compression making disk and utilisation minimal.
Atlas also have full text search and charts building tools to extend log analysis capabilities with some simple clicks.
Thanks
Pavel