Migrate to new document structure in mongo 3.6

Working with Data
1

Hello!
I have to migrate data from a structure

  {
    "_id": "some-id",
    "_class": "org.some.class",
    "number": 1015,
    "timestamp": {"$date": "2020-09-05T12:08:02.809Z"},
    "cost": 0.9200000166893005
  }

to

    "_id": {
      "productId": "some-id",
      "countryCode": "DE"
    },
    "_class": "org.some.class",
    "number": 1015,
    "timestamp": {"$date": "2020-09-05T12:08:02.809Z"},
    "cost": 0.9200000166893005
  }

The change that is in the new document is the _id field is replaced by a complex _id object (productId : String, country : String)
The country field is to be completed for the entire collection with a specific value - DE.
The collection has about 40 million records in the old format and 700k in the new format. I would like to bring these 40 million to this new form. I’m using mongo 3.6, so I’m a bit limited and I’ll probably have to use the aggregate functions to create a completely new collection, and then remove the old one. I will be grateful for help on how to do it - how the query that will do it should look like and how to keep these migrated 700k documents.

2

What I have got so far:

db.productDetails.aggregate(
{$match: {_id: {$exists: true}}},
{$addFields: {"_id": {"productId": "$_id", "country": "DE"}},
{$project: {_id: 1, _class: 1, number: 1, timestamp: 1, cost: 1}},
{$out: "productDetailsV2"}
)

but this solution would only work if I didn’t have 700k documents in the new form.

3

The first smart move would be to migrate to a supported version.

Be aware that the object

_id : { product : 1 , country : 2 }

is not equal to the object

_id : { country : 2 , product : 1 }

despite the fact that

collection.find( { "_id.product" : 1 , "_id.country" : 2 } )

will find the same documents as

collection.find( { "_id.country" : 2 , "_id.product" : 1 } )

Will match all documents. I think you want { “_id.products” : { “$exists” : false } }.

4

quick note: “_id” field is flexible in its data type, however, please try to keep it being an “ObjectId” (or at least simple as GUID, Int, or String).

if you still want to go with your change in the “_id” field, check this one:

[ 
  {$addFields: {
    "_id":{
      $cond:{
        if:{$not:["$_id.productId"]},
        then: {"productId": "$_id", "country": "DE"}},
        else:"$_id"
      }
    }
  }},
 {$out: "productDetails_all_new"}
]

$match is not needed here. $project is also not needed unless you are changing the shape.

this will process all documents, old and new, and replaces all old “_id” fields, so try on a test collection first as you might have parts we are not aware of.