Aggregation lookup not getting the document more than once if I am looking up the same ID

[Emilio_El_Murr]

Hello,

I am working on an audio streaming platform and I am facing a problem where if a user adds the same song twice in a playlist I will only be able to recieve it once,

I am running the following query:

media.aggregate(
    [
        {
            "$match":
                {
                    "uuid":"9ca7b8b4f68348869f05dbcf4e62a560",
                    "type":"playlist"
                }
            },
       {
           "$lookup":
           {
               "from":"media",
               "localField":"tracks._id",
               "foreignField":"_id",
               "as":"tracks"
          }
        }
    ]
)

This is the data that is present in my playlist before the aggregation:

{
"_id": "62baa8249f19e6ca929191f5",
            "title": "This is a playlist",
            "tracks": [
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                },
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    }
                }
            ],
            "type": "playlist"
}

My goal is to get all the tracks even though they are the same, I would like them aggregated as multiple objects but I am getting the following response when i execute the query:

            "_id": "62baa8249f19e6ca929191f5",
            "title": "This is a playlist",
            "tracks": [
                {
                    "_id": {
                        "$oid": "62bd5bb68fce34575360669c"
                    },
                    "album": {
                        "_id": {
                            "$oid": "62bd63c88fce34575360669d"
                        },
                        "thumbnail": " https://i.scdn.co/image/ab67616d00001e02a9956b47dc867ce769c7841f"
                    },
                    "artists": [
                        {
                            "_id": {
                                "$oid": "62bd69c68fce3457536066a1"
                            }
                        }
                    ],
                    "disc_number": 1,
                    "duration": 166.64,
                    "explicit": false,
                    "is_playable": true,
                    "name": "About A Girl ( New Release )",
                    "popularity": 73,
                    "track": true,
                    "track_number": 2,
                    "type": "track",
                    "uuid": "55yvzYuvJYG2RUEnMK78tr",
                    "audio": {
                        "_id": {
                            "$oid": "62bd8b518fce3457536066b1"
                        },
                        "duration": 483000
                    },
                    "is_new_release": true,
                    "is_promoted": true,
                    "title": "hey"
                }
            ],
            "type": "playlist"
}

[Pavel_Duchovny]

Hi @Emilio_El_Murr ,

Well to do what you want the aggregation might get more complex. In general the best guidance is in the playlist document store any information that is vital for the presented songs in the playlist and not only ideas to avoid complex self joins.

However, if you still like to try my workaround pipeline you are welcome:

[{
 $match: {
  _id: '62baa8249f19e6ca929191f5',
  type: 'playlist'
 }
}, {
 $unwind: {
  path: '$tracks'
 }
}, {
 $lookup: {
  from: 'media',
  localField: 'tracks._id',
  foreignField: '_id',
  as: 'tracks'
 }
}, {
 $facet: {
  rootObj: [
   {
    $limit: 1
   }
  ],
  tracks: [
   {
    $group: {
     _id: '$_id',
     tracks: {
      $push: {
       $first: '$tracks'
      }
     }
    }
   }
  ]
 }
}, {
 $replaceRoot: {
  newRoot: {
   $mergeObjects: [
    {
     $first: '$rootObj'
    },
    {
     $first: '$tracks'
    }
   ]
  }
 }
}]

[steevej (Steeve Juneau)]

This is some kind of optimization that I came to appreciate. By doing it like this, duplicated values are $lookup-ed up only once save processing time on the server. It also preserve bandwith since duplicate resulting documents are not sent over the wire.

An alternative to the $unwind/$group would be $set stage where you use $map to complete the source array with the resulting array. That would negate the optimization supplied by default. But that’s fine because it would be your choice and others, like me would keep benefiting from the optimization. I prefer to do this kind of data structure cosmetic at the application level as it is easier to scale. Doing the final matching and sending duplicate data from the server affects all users. Doing the final matching in the client code, and even in the front end, only affect the ones with duplicate.