Scenario: We have an operational database
ops_db on a replica set that retains the data for 3 months and purges anything beyond that. Now, we have a requirement from Analytics team to provide them with a solution to retain the data for 5/7 years for analytical purpose.
I can think of using other platforms/tools to accomplish this. For example, use Kafka connect and
ops_db will produce data into a topic, and analytical database
bid_db will consume data.
bid_db will be a different database sitting on the same mongo cluster. However, this comes with additional cost for a new platform/tool which we want to avoid at this stage.
So, with no extra platforms/tools how can we achieve the solution of continuously copying data from
bid_dbthat resides on the same mongo cluster?
Note: We don’t want to increase retention policy on
ops_db, so looking to create
Also open to other approaches/suggestions that doesn’t require to purchase new platforms/tools.