Frequent updates on nested arrays with deep merging

Hi everyone!

I wanted to ask community what is the preferable architecture / way of storing data which is frequently updated?

For example I have an user collection and every user has his in-game quests.
Quests have their progression which will be very often modified / updated.
Exemplary user would look like this:

{

  "_id": { "$oid": "xxxxxxxxxxxxxxxxxxxxxxxx" },

  "user_id": 1,

  "quests": [

    {

      "startDate": { "$date": "2021-07-18T13:30:36.617Z" },

      "endDate": { "$date": "2021-07-18T13:30:46.617Z" },

      "reward": {

        "_id": { "$oid": "60f4102d59380d229455b530" },

        "xp": 300,

        "currency": 200

      },

      "conditions": [

        {

          "type": "play",

          "details": {

            "_id": { "$oid": "60f42cfc5a79ef478c5938ab" },

            "count": 10,

            "winAtleast": 5,

            "loseMax": 2

          },

          "progress": {

            "gamesInRow": ["win", "lose", "lose"]

          }

        }

      ]

    }

  ]

}

At first I thought nested arrays / embedded docs are the way to go, because there is one to many relation, where every user has multiple quests but they are unique for them.

But when I started programming an API for it to update this structure I got a big problem with deep merge (javascript driver). I want to have a nice way to update progress fields and these updates will be very frequent. I read about arrayFilters and dot notations to make this possible but the code with this design is very messy.

The main problem for me now is to transform object which looks like user object but with updated fields on every “nesting level” to update / updateMany function with all those arrayFilters etc.
I know how to change single field but I would like to have one resuable function with whole object as parameter to update only these fields which are changed not replace whole user everytime.

Maybe I can achieve this by breaking this structure into multiple collections? Still I would love to know preferable and efficient way to do frequent operations on nested objects / arrays.

I hope everyone of You will have a great day!

Hi @Loud_Silence ,

Welcome to MongoDB community.

The consideration you raised are exactly the reason why relationship in MongoDB does not have a thumbrule to implement.

If the relationship is one to many I would suggest using a hybrid approach perhaps:

  1. Immutable or less frequently updated fields will be in the users embedded array.
  2. Frequently updated fields with any search criteria (userId, questId etc) will be separated into quests collection.

You will probably prefer to run 2 queries to gather a single user view but possibly use a $lookup if no other choice.

I would recommend reading:

Please let me know if this make sense to you.

Thanks
Pavel

Thank you very much for your response!

I read both posts and some other recommendations, they’re very helpful in learning mongo best practices.
I even decided that I will try the hybrid approach You suggested.

Now my only concerns are that even after breaking up some nested data to another collection. Most of it will still be quite often updated and my deep merging problems remains if its nested couple of times.

For example progress fields will be updated very often but after reading those articles I feel like dividing progress object of every quest’s condition in separate collection is kind of anti-pattern, because I will end up with too many collections (User, conditions, details, progress). So there will be a problem when I will need to read everything. But if I leave user collection and make another one only for conditions it’s getting better but still could end up heavily nested, especially if progress or details will have nesting in it.

I know that I’m asking a lot for an simple advice but is there any guidance on how to make an API for deep nested structures in any programming language / mongo drivers?

Currently I’m using mongoose and I ended up with recursive function which will translate whole object to flat object with fields made with “dot notation” something like this: https://www.npmjs.com/package/dot-object

Still I’m feeling like I’m overengineering this case, I thought cases like that are very common things in MongoDB but most information I find about it tells how to update specific field / specific object in single array and they’re using mainly hardcoded stuff.

Anyway Your response helped me a lot in understanding how MongoDB works! Thank You once more!

Hi @Loud_Silence ,

Thanks for the kind words.

I will try to help you based on the provided information and the common sense of gaming applications although I am not aware of the full requirements or query arsenal of your use case (and I assume neither are you fully). The good thing about MongoDB flexible schema is that you can change it as time go and if you do not hit an antipattern you should be ok for your initial design.

What I’ve suggested is to split the presented data into 2 (maybe 3) collections, but possibly the relationship will still be one to many.

Users collection

{

  "_id": { "$oid": "xxxxxxxxxxxxxxxxxxxxxxxx" },

  "user_id": 1,

  "quests": [

    {

      "startDate": { "$date": ... }, 
      questId : "xxx"
},
    {

      "startDate": { "$date": ... }, 
      questId : "yyy"
}
...
]

In this collection a user stores user data and pushes a quest with its start date and generated id (could be w new ObjectId() into the users collection.

Quests collection

{
     "questId" : "xxx" ,
     "userId" : { "$oid": "xxxxxxxxxxxxxxxxxxxxxxxx" },
      "startDate": { "$date": "2021-07-18T13:30:36.617Z" },

      "endDate": { "$date": "2021-07-18T13:30:46.617Z" },

      "reward": {

        "_id": { "$oid": "60f4102d59380d229455b530" },

        "xp": 300,

        "currency": 200

      },

      "conditions": [

        {

          "type": "play",

          "details": {

            "_id": { "$oid": "60f42cfc5a79ef478c5938ab" },

            "count": 10,

            "winAtleast": 5,

            "loseMax": 2

          },

          "progress": {

            "gamesInRow": ["win", "lose", "lose"]

          }

        }

      ]

    }

You can split reward/condition/progress into seperate documents in the same collection by adding a type : .... field and indexing that.

In general you should probably index {,userId : 1, questId :1 } to optimally locate users quests.

Now operating nested arrays can be done via arrayFilters which is the straight forward approach : https://docs.mongodb.com/manual/reference/operator/update/positional-filtered/

You will need to place the filter array dynamically and use those fields when pushing or setting fields, but when the object has only one quest this filtering becomes less painful even if you nesting level is 2 levels below. If your nesting is more than 2 levels maybe consider adding an extension 3rd collection to hold this extra deep data and update it.

Another option is to use pipeline aggregation updates with a $map for example to rebuild an array by traversing it , search the forum for update with $map there are several examples:

Let me know if this makes sense.

Best regards,
Pavel

Thank You @Pavel_Duchovny !

I will go in this direction and “battle-test” it with every kind of updates also I’ll checkmark this as a solution, and hopefully this will work nicely with my cases!

Have a nice day and thanks once more!

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.