Aggregation not getting more than 1 of the document even though I have duplicated entries on purpose

Working with Data
1

Hey there,

I am running into an issue and not sure how to proceed. I have an aggregation pipeline that links a couple collections based on some field id in the document. However one of those field (Files) has multiple entries that are similar, like so:

/**

  • Paste one or more documents here
    */
    {
    “name”: “Miniature Storage (Marvel United Multiverse)”,
    “sku”: “MS-MUM”,
    “files”: [
    {
    “file”: {
    “$oid”: “66b2200aef77b9fedbaa617b”
    },
    “requiredCount”: 0
    },
    {
    “file”: {
    “$oid”: “66b2200aef77b9fedbaa617c”
    },
    “requiredCount”: 0
    },
    {
    “file”: {
    “$oid”: “66b2200aef77b9fedbaa617d”
    },
    “requiredCount”: 0
    },
    {
    “file”: {
    “$oid”: “66b2200aef77b9fedbaa617e”
    },
    “requiredCount”: 0
    },
    {
    “file”: {
    “$oid”: “66b2200aef77b9fedbaa617e”
    },
    “requiredCount”: 0
    },
    {
    “file”: {
    “$oid”: “66b2200aef77b9fedbaa617e”
    },
    “requiredCount”: 0
    },
    {
    “file”: {
    “$oid”: “66b2200aef77b9fedbaa6180”
    },
    “requiredCount”: 0
    }
    ],
    “materialType”: “Wood”,
    “machineType”: “standard”
    }

When the data is returned it omits the duplicated entries, I however, need those returned to be able to be rendered properly in the UI.

Any ideas ?

Thank you.

2

We have no ideas about what is wrong with your aggregation if you do not share the said aggregation.

Please read about code snippet and share your sample documents, your aggregation and the expected result in a format that we can experiment with.

Even better you may create a playground that we can start experimenting right away.

3

Ok my bad, here’s the playground with the data.

Thanks again!

4

In the playground you

{ $lookup": { "from": "Materials", ... } }

but there is no such collection in the sample data.

5

While looking closer at your issue, I think that what you are experimenting is a welcome optimization where $lookup is not performed on multiple items.

Doing multiple $lookup over the same item and returning multiple time the same sub-document is potentially harmful.

First, let say that you do not need to use let with array lookup. Simple $lookup like shown here is sufficient to gather all the information.

It is then the role of your application to find the correct sub-document from files_lookup and material_lookup to reconstruct the whole things with duplicated.

If you insist in having the server to feed duplicates, then you could write $addFields stage that uses $map from files with $indexOfArray and $arrayElemAt to build your tree of duplicate objects. But more work and more data sent by the server means less free CPU and less free network bandwidth to serve other client.

In both case, it is the same code logic, mapping an array to another one by looking up complex object in another array. I think it is easier to scale that logic at the application level rather than the data level.

1 Like
6

Thanks thats exactly what i needed to do! Wasn’t aware lookup stripped/not pull the duplicates but makes sense as an optimization process.

Thanks again!

1 Like
7

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.