Hi EveryOne,
Actually, I need to configure live replication from AWS Postgresql to Atlas AWS hosted MongoDB but this is not one time activity so replication should be available all the time. so I will use postgresql and mongodb as well so we able to move data’s from postgresql to mongodb By using relation migrator with Continous approach not as snapshot approach. and I tested this with sample data’s but will it work in production server? because I read relational migrator not recommended for indefinite moving data’s in mongodb blog. so I want to know will we use relational migrator for this scenario? around 4 lakhs data will come to postgre per day so I need to clarify this?
Thanks & Regards
Aravind
Data Engineer
Good morning, welcome to the MongoDB community.
I would try to do it differently. Maybe using Kafka or some other streaming platform, taking the CDC and saving it to MongoDB. Maybe Stream Processing will help you.
I really don’t know if Migrator is the best tool to use in this case. Let me know what you think ;D
I’m a beginner in MongoDB but I read about the Atlas Stream Processing so as of my knowledge Atlas Stream Processing (ASP) is primarily designed for real-time event processing within MongoDB Atlas. It allows you to capture and process changes that occur within your MongoDB collections and take actions based on those changes, such as sending notifications, performing analytics, or triggering other events. but in my case I need configure replication from PostgreSQL to MongoDB. initially all data should move from PostgreSQL to Atlas mongodb then if any data newly coming in PostgreSQL then those data should move to Atlas Mongodb and also if any previous data updated those update also should replicate in Atlas mongodb so in this scenario will it work? But I already tried Relational migrator with continous sink. it’s working but I read this not recommended for indifinte running so this is my concern. and will it able to move data contrinously from PostgreSQL to Atlas MongoDB by using Atlas Stream Processing?
In this case, you can use CDC to do this from PG to Atlas. Use some connector, for example debizium and use the mongodb connector for kafka.
Example: How to Implement Change Data Capture for MySQL and Postgres | by Lewis Gavin | Rockset | Medium