Update existing array of objects in the document dynamically via matching with data set of new array of objects

I have a document that comprises of the following schema (this collection is meant to save user’s history in completing a section). The following schema assumes that a section may have 1-M topics with each of their respective topics

{
  "_id": "someId",
  "history": [
    {
      "sectionId": "section0",
      "topicId": 0,
      "status": 1,
      "completedDate": "2/9/2022 10:46:34 AM"
    },
    {
      "sectionId": "section0",
      "topicId": 1,
      "status": 0
    },
    {
      "sectionId": "section0",
      "topicId": 2,
      "status": 1,
      "completedDate": "2/9/2022 8:24:35 AM"
    },
    {
      "sectionId": "section0",
      "topicId": 3,
      "status": 0
    },
    {
      "sectionId": "section0",
      "topicId": 4,
      "status": 0
    },
    {
      "sectionId": "section1",
      "topicId": 0,
      "status": 0
    },
    {
      "sectionId": "section1",
      "topicId": 1,
      "status": 0
    },
    {
      "sectionId": "section2",
      "topicId": 0,
      "status": 0
    },
    {
      "sectionId": "section2",
      "topicId": 1,
      "status": 0
    }
  ]
}

In above example, status 0 refers to uncompleted topic within a section, whereas status 1 refers to completed topic with completedDate as well.
This document is generated dynamically after a user subscribes to an online quiz preparation (quiz-preparation collection) whose schema is like this

{
  "_id": "someID",
  "quizContents": [
    {
      "sectionId": "section0",
      "topicName": "ABC",
      "topics": [
        {
          "topicId": 0
        },
        {
          "topicId": 1
        },
        {
          "topicId": 2
        },
        {
          "topicId": 3
        },
        {
          "topicId": 4
        }
      ]
    },
    {
      "sectionId": "section1",
      "topicName": "DEF",
      "topics": [
        {
          "topicId": 0
        },
        {
          "topicId": 1
        }
      ]
    },
    {
      "sectionId": "section2",
      "topicName": "GHI",
      "topics": [
        {
          "topicId": 0
        },
        {
          "topicId": 1
        }
      ]
    }
  ]
}

Now suppose if quiz is updated (new topics or sections are added), or some sections are removed (suppose section 2), then user history shall reflect that as well, meaning if any section is deleted then it shall be deleted from user history (in first example) as well. Whereas if any section is updated with new topics then user history array shall have new topic as well with status set to 0.
Now when I update quiz sections, I am passing new array of objects to user history schema, (I am aware how to achieve this via JavaScript) but is there a way to achieve this using Mongodb Native Driver in Node.js

Hi @Daniyal_Khan ,

If I understand correctly you are looking into passing just Delta of an array and updating just that data.

This can be achieved with arrayFilter updates, where you can conditionally update certain array elements in a document:

For example you can pass on the updated section ids and the respectful status and in the update commands run multiple updates in $set:

update({_id : ... },{$set : { "history.$[section1].status" : 1, "history.$[section2]. completedDate" : new Date()} , { arrayFilters: [ { "section1": sectionID1,  "section2": sectionID2} } ] })

Now obviously the above is an untested psedocode but the idea is similar for pulls and pashes to an array.

Having said that, many concurrent operations on arrays is not a very healthy thing so I would still test it against building the history array on client side and updating it as a whole

Thanks
Pavel

I am looking for the following desired results after update (I am using examples given above before update)

If contents within course is updated, like given below

{
      "_id": "someID",
      "contents": [
        {
          "sectionId": "section0",
          "topicName": "ABC",
          "topics": [
            {
              "topicId": 0
            },
            {
              "topicId": 1
            },
            {
              "topicId": 2
            },
            {
              "topicId": 3
            },
            {
              "topicId": 4
            }
            {
              "topicId": 5
            }
          ]
        },
        {
          "sectionId": "section2",
          "topicName": "GHI",
          "topics": [
            {
              "topicId": 0
            },
            {
              "topicId": 1
            }
          ]
        }
      ]
    }

then userHistory schema shall also change accordingly

{
      "_id": "someId",
      "history": [
        {
          "sectionId": "section0",
          "topicId": 0,
          "status": 1,
          "completedDate": "2/9/2022 10:46:34 AM"
        },
        {
          "sectionId": "section0",
          "topicId": 1,
          "status": 0
        },
        {
          "sectionId": "section0",
          "topicId": 2,
          "status": 1,
          "completedDate": "2/9/2022 8:24:35 AM"
        },
        {
          "sectionId": "section0",
          "topicId": 3,
          "status": 0
        },
        {
          "sectionId": "section0",
          "topicId": 4,
          "status": 0
        },
        {
          "sectionId": "section0",
          "topicId": 5,
          "status": 0
        },
        {
          "sectionId": "section2",
          "topicId": 0,
          "status": 0
        },
        {
          "sectionId": "section2",
          "topicId": 1,
          "status": 0
        }
      ]
    }

Is this achievable via arrayFilters?
As it requires dynamic sessionId changes

Dynamic sessionId changes? What do you mean?

Can you show me the update command you run on the course collection?

I have yet to do anything, the idea is that mongodb method updateMany will take a new array (consider updated course contents array), and match with exisitng array elements within document and update that accordingly

Update userCourseHistory method

  static async updateUsersHistory(courseId, history) {
    try {
      const resetUserCourseHistory = await userCourseHistory.updateMany(
        {
          courseId: courseId,
        },
        {
          //Here update query will take an array as input (input given below this function), and then apply filter or map on existing array elements (example given below) dynamically
        }
      );

      return resetUserCourseHistory;
    } catch (err) {
      console.error(
        `Error occurred in updating all users course history, ${err}.`
      );
      return null;
    }
  }

Input Array

{
      "_id": "someID",
      "contents": [
        {
          "sectionId": "section0",
          "topicName": "ABC",
          "topics": [
            {
              "topicId": 0
            },
            {
              "topicId": 1
            },
            {
              "topicId": 2
            },
            {
              "topicId": 3
            },
            {
              "topicId": 4
            }
            {
              "topicId": 5
            }
          ]
        },
        {
          "sectionId": "section2",
          "topicName": "GHI",
          "topics": [
            {
              "topicId": 0
            },
            {
              "topicId": 1
            }
          ]
        }
      ]
    }

Prior userCourseHistory

{
  "_id": "someId",
  "history": [
    {
      "sectionId": "section0",
      "topicId": 0,
      "status": 1,
      "completedDate": "2/9/2022 10:46:34 AM"
    },
    {
      "sectionId": "section0",
      "topicId": 1,
      "status": 0
    },
    {
      "sectionId": "section0",
      "topicId": 2,
      "status": 1,
      "completedDate": "2/9/2022 8:24:35 AM"
    },
    {
      "sectionId": "section0",
      "topicId": 3,
      "status": 0
    },
    {
      "sectionId": "section0",
      "topicId": 4,
      "status": 0
    },
    {
      "sectionId": "section1",
      "topicId": 0,
      "status": 0
    },
    {
      "sectionId": "section1",
      "topicId": 1,
      "status": 0
    },
    {
      "sectionId": "section2",
      "topicId": 0,
      "status": 0
    },
    {
      "sectionId": "section2",
      "topicId": 1,
      "status": 0
    }
  ]
}

that will result in update userCourseHistory as following

{
      "_id": "someId",
      "history": [
        {
          "sectionId": "section0",
          "topicId": 0,
          "status": 1,
          "completedDate": "2/9/2022 10:46:34 AM"
        },
        {
          "sectionId": "section0",
          "topicId": 1,
          "status": 0
        },
        {
          "sectionId": "section0",
          "topicId": 2,
          "status": 1,
          "completedDate": "2/9/2022 8:24:35 AM"
        },
        {
          "sectionId": "section0",
          "topicId": 3,
          "status": 0
        },
        {
          "sectionId": "section0",
          "topicId": 4,
          "status": 0
        },
        {
          "sectionId": "section0",
          "topicId": 5,
          "status": 0
        },
        {
          "sectionId": "section2",
          "topicId": 0,
          "status": 0
        },
        {
          "sectionId": "section2",
          "topicId": 1,
          "status": 0
        }
      ]
    }

Sorry I don’t understand the logic of the update…

Can you show how individual input in the initial document impacting specific sections in this history…

Let me explain it step by step (Note: The schema in question is inherited and might not be ideal due to the complexity of fluid user requirements).

This is the schema of a course (for the sake of brevity, I am skipping other details)

{
  "_id": "courseID",
  "title": "Demo",
  "quizContents": [
    {
      "sectionId": "section0",
      "topicName": "ABC",
      "topics": [
        {
          "topicId": 0
        },
        {
          "topicId": 1
        },
        {
          "topicId": 2
        },
      ]
    },
    {
      "sectionId": "section1",
      "topicName": "DEF",
      "topics": [
        {
          "topicId": 0
        },
        {
          "topicId": 1
        }
      ]
    },
    {
      "sectionId": "section2",
      "topicName": "GHI",
      "topics": [
        {
          "topicId": 0
        },
        {
          "topicId": 1
        }
      ]
    }
  ]
}

This is modelled after the course section in udemy, where sectionId (topicName) refers to outer section (like Chapter 0: Introduction and Aggregation Concepts), and each topics nested within a section are the chapters sub-topics.

Now when a user purchases this course, a new document is created in userHistory collection, that dynamically calculates total topics and inserts the following document

{
  "_id": "someId",
  "courseId": "courseID",
  "userId": "userID",
  "history": [
    {
      "sectionId": "section0",
      "topicId": 0,
      "status": 0,
    },
    {
      "sectionId": "section0",
      "topicId": 1,
      "status": 0
    },
    {
      "sectionId": "section0",
      "topicId": 2,
      "status": 0,
    },
    {
      "sectionId": "section1",
      "topicId": 0,
      "status": 0
    },
    {
      "sectionId": "section1",
      "topicId": 1,
      "status": 0
    },
    {
      "sectionId": "section2",
      "topicId": 0,
      "status": 0
    },
    {
      "sectionId": "section2",
      "topicId": 1,
      "status": 0
    }
  ]
}

Notice, that each section and its corresponding topic is assigned status 0 (that means that user has not completed this section). The rest are 0 due to allowing user to click any section, so that any section can be updated (section 2 can be updated without setting section0, or section1 status as complete).

Now when a user has completed a section (first topic within first section) then its updated as follows

{
  "_id": "someId",
  "courseId": "courseID",
  "userId": "userID",
  "history": [
    {
      "sectionId": "section0",
      "topicId": 0,
      "status": 1,
      "completedDate": "2/9/2022 10:46:34 AM"
    },
    {
      "sectionId": "section0",
      "topicId": 1,
      "status": 0
    },
    {
      "sectionId": "section0",
      "topicId": 2,
      "status": 0,
    },
    {
      "sectionId": "section1",
      "topicId": 0,
      "status": 0
    },
    {
      "sectionId": "section1",
      "topicId": 1,
      "status": 0
    },
    {
      "sectionId": "section2",
      "topicId": 0,
      "status": 0
    },
    {
      "sectionId": "section2",
      "topicId": 1,
      "status": 0
    }
  ]
}

Now this was working fine but a new problem occurred recently, that if course contents is updated then all user history shall also be updated accordingly.

For updating course contents, I take the data from client-side, dynamically convert it to arrays and pass it to MongoDB for updating using $set operator that simply overwrites existing arrays in place. Like the following

New Course Contents

{
  "_id": "courseID",
  "title": "Demo",
  "quizContents": [
    {
      "sectionId": "section1",
      "topicName": "DEF",
      "topics": [
        {
          "topicId": 0
        },
        {
          "topicId": 1
        }
      ]
    },
  ]
}

Notice that both section0 and section2 were removed (sectionIds are not numbered as user history data cannot be changed). I provide this new array to course DAO, and it replaces existing array with new array as provided above.

The problem is that I want to provide an array to userHistory document, in order for MongoDB to iterate data array provided as input, and simultaneously update all existing userHistorys, basically I want to achieve the following thing.

Match existing array A1 with new array dataset A2, and update existing array A1 after mapping with A2.

The desired output is as follows

{
  "_id": "someId",
  "courseId": "courseID",
  "userId": "userID",
  "history": [
    {
      "sectionId": "section1",
      "topicId": 0,
      "status": 0
    },
    {
      "sectionId": "section1",
      "topicId": 1,
      "status": 0
    },
  ]
}

Now userHistory does not contain section0, and section2 respectively. This is a contrived example, as in real-world, it can also contain new topics within a section as well.

Is this possible in MongoDB either via aggregation framework or any other step?

Thanks for your time, its appreciated!