Populate a dev db from prd

Working with Data
1

Hi there,
to populate my test projects in Atlas I currently restore an old database whenever I feel the need of “clean” data. The production environment often grows and the database get’s to big to be restored in a small dev environment. Which option would be best to populate my dev DB?

  • Can I run aggregations in one project which writes to an other project?
  • Or do I need to setup views which I dump and restore (actual the classical way)?
  • Or is there a smarter process?

Thanks a lot for any suggestion
Michael

2

Hi @michael_hoeller,

What is your ultimate goal ?

To get all collections with partial data into dev ?

Or some collections with all data ?

Thanks
Pavel

3

Hi @Pavel_Duchovny

the first: to get all collections with partial data into dev. The get consistent (partial) data to dev some relations need to be fulfilled.

Michael

4

@michael_hoeller,

Sounds like a dump with query parameter to export specific documents from all collections.

Another possibility is to use a fetch and bulk insert via a script considering your relationships logic.

If the clusters are in the same project you can use Realm functions to copy data between them :wink:

Thanks
Pavel

5

@Pavel_Duchovny

I thought about this just this is a simple query on one collection, to keep consistency I need to cover relations to other collections. Just to maintain queries is do able but complex, also often some data needs to be anonymized when moved from prd to somewhere.

They are not, I’d assume due to access rights that in most cases prd and dev will be in different projects.
This would be interesting to if possible.

Same as with realm.

In the past I mainly stripped down a local copy and keep a dump of this db as blue print for quickly setup dev dbs. This is surely not an ideal process, and the above mentioned issues drove me to write this post - hoping that I missed a cool trick.