MongoDB ChangeStream watch for multiple nested property changes

MongoDB document structure that I need to watch for changes is something like below:

{
    "_id": 1111,
    "staticName": {
      "dynamicName_1": {
        "dynamicName_2": {
          "staticName_2": "static name number 2",
          "staticName_3": "static name number 3",
          "staticArrayToTrackChanges": [
            {
             "static_do_not_track_1":1, 
             "static_do_not_track_2":2, 
             "static_track_this_for_changes":111
            },
            {
             "static_do_not_track_1":3, 
             "static_do_not_track_2":4, 
             "static_track_this_for_changes":112
            }
           ]            
        }
      }
    }    
}

Every property with keyword static inside name is property that has static (identical) name inside every document.

Properties dynamicName_1 and dynamicName_2 are Dictionary keys that have dynamic values and are different for each document.

I need to be able to track changes for property static_track_this_for_changes . Every document will have this property, but under different dynamicName_1 and dynamicName_2 keys.

Also updateDescription.updatedFields of ChangeStreamDocument should contain only staticArrayToTrackChanges object that contains changed static_track_this_for_changes .

Any help toward right solution for this is very appreciated :slight_smile:

Hi @Marko_Saravanja and welcome in the MongoDB Community :smiley: !

To my knowledge, it’s not possible to track only these fields in subdocuments if you don’t know the path to these. If by any chance, dynamicName_1 and 2 are actually limited to a few values, you could use $or and $exists all the different combinations of X.Y.staticName_1.

But I think the easiest path here it to change the model and avoid dynamic names in fields. It’s usually a nightmare to deal with dynamic fields in the backend.

Would it be possible to update the document to something like this for example?

{
    "_id": 1111,
    "staticName": {
      "staticStuff_1": {
        "type": "dynamicName_1",
        "staticStuff_2": {
          "someNumber": "dynamicName_2",
          "staticName_2": "static name number 2",
          "staticName_3": "static name number 3",
          "staticArrayToTrackChanges": [
            {
             "static_do_not_track_1":1, 
             "static_do_not_track_2":2, 
             "static_track_this_for_changes":111
            },
            {
             "static_do_not_track_1":3, 
             "static_do_not_track_2":4, 
             "static_track_this_for_changes":112
            }
           ]            
        }
      }
    }    
}

Then it’s trivial to use a $exists on staticName.staticStuff_1.staticStuff_2.staticName_1 and staticName.staticStuff_1.staticStuff_2.staticName_1 with $or.

I’m trying to ask around if someone has a smarter / better idea but no chance so far.

Cheers,
Maxime.

1 Like

I assume you know how many levels deep these static fields are?

You could use $objectToArray expression to convert a document into array of key value pairs - you can now match on the “value” part - though the document you show is quite complex so you would need to apply this technique a couple of levels deep…

Asya

1 Like

Hi Maxime, thank you for your welcome.
We have agreed to remove dynamic names from mongo schema, and use static ones.
Also, we have removed one level of nesting, and merged two dynamics into one object (we now have more objects in array, but that is fine), so now object is something like

{
    "_id": 1111,
    "staticName": [
    {
        "type1": "dynamic_type1_1",
        "prop1": "some_value",
        "type2": "dynamic_type2_2"
    },
    {
        "type1": "dynamic_type1_1",
        "prop1": "some_value",
        "type2": "dynamic_type2_3"
    },
    {
        "type1": "dynamic_type1_2",
        "prop1": "some_value",
        "type2": "dynamic_type2_4"
    }
    ]
}

Issue that I am having now is as following:

  • I must querry documents to some filter and that is mostly where type1 = some value. That part is OK.
  • Result set contains multiple objects where type1 = some value. They all have different type2 values.
  • Then, I must project matched results to only include those objects from array that are in match condition.
  • Example: search condition is type1 = dynamic_type1_1. Document with id 1111 matches filter. From that document, inside array staticName, i should project results to include only those objects where type1 = dynamic_type1_1. In document above, that means I should project only first two objects from array staticName and exclude third one where type1 = dynamic_type1_2.

I do not know is this the right place to continue our discussion, so feel free to point that out if necessary.

Cheers,
Marko

Hi Asya,
appreciated very much your time invested in providing solution, but it did struck me the issues that we would have while fighting with dynamic names on backend side, so we decided to remove dynamics completely.

Cheers,
Marko

You can do this using $filter expression.

1 Like

Excellent news that it was possible to change the previous document model.
I wrote a little script to show you the result using $filter @Marko_Saravanja.

db.coll.drop();
db.coll.insertOne({
    "_id": 1111,
    "staticName": [
    {
        "type1": "dynamic_type1_1",
        "prop1": "some_value",
        "type2": "dynamic_type2_2"
    },
    {
        "type1": "dynamic_type1_1",
        "prop1": "some_value",
        "type2": "dynamic_type2_3"
    },
    {
        "type1": "dynamic_type1_2",
        "prop1": "some_value",
        "type2": "dynamic_type2_4"
    }
    ]
});

db.coll.createIndex({'staticName.type1': 1});

db.coll.aggregate([
  {
    '$match': {
      'staticName.type1': 'dynamic_type1_1'
    }
  }, {
    '$project': {
      'staticName': {
        '$filter': {
          'input': '$staticName', 
          'as': 'value', 
          'cond': {
            '$eq': [
              '$$value.type1', 'dynamic_type1_1'
            ]
          }
        }
      }
    }
  }
]);

I took the liberty to add a little index in the mix that you need ;-). I bet you will probably need the same on type2 :slight_smile:.

Result of this little aggregation:

[
  {
    _id: 1111,
    staticName: [
      {
        type1: 'dynamic_type1_1',
        prop1: 'some_value',
        type2: 'dynamic_type2_2'
      },
      {
        type1: 'dynamic_type1_1',
        prop1: 'some_value',
        type2: 'dynamic_type2_3'
      }
    ]
  }
]

Another naive approach would be to use $match + $unwind + $match + $group on _id with $push to recreate the array but it’s just a lot more processing…

Cheers,
Maxime.

1 Like

I think by “naive” you meant to say “really bad”, “inefficient” and “definitely unadvised” :slight_smile:

2 Likes

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.