Populate Array of ObjectIds While Preserving Order

Working with Data
,
1

Is it possible to populate nested arrays of object ids while preserving order? I understand that lookup is essentially random but is there a way to use the initial array of object ids to set the order for the populated ids after the lookups?

My initial model looks like this:

{
    ...,
    guides: [
        // array of object ids
    ]
}

and the final would look like

{
    ...,
    guides: [
        {
            ...,
            sections: [
                {
                    ...
                }
            ]
        }
    ]
}

Current pipeline - works but items are unordered.

    {
      $match: {
        $expr: {
          $eq: ["$name", cookbookName],
        },
      },
    },
    {
      $lookup: {
        from: "guides",
        localField: "guides",
        foreignField: "_id",
        as: "guides",
      },
    },
    {
      $unwind: {
        path: "$guides",
      },
    },
    {
      $lookup: {
        from: "sections",
        localField: "guides.sections",
        foreignField: "_id",
        as: "guides.sections",
      },
    },
    {
      $group: {
        _id: "$_id",
        name: {
          $first: "$name",
        },
        avatar_url: {
          $first: "$avatar_url",
        },
        banner_url: {
          $first: "$banner_url",
        },
        game: {
          $first: "$game",
        },
        preview: {
          $first: "$preview",
        },
        roles: {
          $first: "$roles",
        },
        streams: {
          $first: "$streams",
        },
        guides: {
          $push: "$guides",
        },
      },
    },
2

I guess that for efficient reasons it has been decided to not guaranty the order of the localField array.

I wrote I guess because I am not too sure it is the reason but it makes sense to me.

1 - it is more efficient to return a document that is already in RAM if it matches rather than flushing it out and the reread it later to return it in the correct order.

2 - it is more efficient to NOT return duplicate documents when the source array mentioned it multiple times.

3 - they could do the lookup more efficiently by just doing a find in the looked up collection using a simple $in query with the localField array as argument, and a find does not sort the document and does not return duplicates.

So it is simpler, more efficient and sufficient for most use-case to return an array without a specific order. And with aggregation you can add stages to get the order you want with $sortArray, sometimes you would use $map on the source array.

In your case, your first localField array is guides and you do $unwind the result. So in your case to preserve the original guides order, you could $unwind before the $lookup rather than $unwind after.

1 Like
3

Hello @steffan, Welcome to the MongoDB community forum,

It will result documents in natural/stored order from the lookup collection.

You can do it by using $map and $filter operators after $lookup, but I can see you are doing $unwind and then the $group stage, also the $group stage does not preserve the order of documents!

I think you don’t need to $unwind here, see the below pipeline,

  {
    $lookup: {
      from: "guides",
      localField: "guides",
      foreignField: "_id",
      as: "guides_arr"
    }
  },
  {
    $lookup: {
      from: "sections",
      localField: "guides_arr.sections",
      foreignField: "_id",
      as: "sections_arr"
    }
  },
  {
    $addFields: {
      guides_arr: {
        $map: {
          input: "$guides_arr",
          as: "g",
          in: {
            $mergeObjects: [
              "$$g",
              {
                sections: {
                  $map: {
                    input: "$$g.sections",
                    as: "s",
                    in: {
                      $first: {
                        $filter: {
                          input: "$sections_arr",
                          cond: { $eq: ["$$s", "$$this._id"] }
                        }
                      }
                    }
                  }
                }
              }
            ]
          }
        }
      }
    }
  },
  {
    $addFields: {
      guides: {
        $map: {
          input: "$guides",
          as: "g",
          in: {
            $first: {
              $filter: {
                input: "$guides_arr",
                cond: { $eq: ["$$g", "$$this._id"] }
              }
            }
          }
        }
      }
    }
  },
  {
    $unset: [
      "guides_arr",
      "sections_arr"
    ]
  }

Playground

4 Likes
4

Wow thank you so much, this was amazing!

5

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.