Querying Twitter Styled Schemas

MongoDB Atlas
, ,
1

Hi! I’m working on a Twitter clone and having some trouble with my aggregation for fetching a user feed’s tweets. My DB has two collections, one for tweets and one for users.

User Model (irrelevant info removed)

_id: ObjectId,
tweets: [tweetObjectId1, tweetObjectId2, ...],
following: ['user1, 'user2, ...]

For a start, I’m trying to get the user’s as well as their following’s tweets in one array. This is my first time using aggregate and this is what I have so far:

db.users.aggregate([
    <!-- Get document of our user -->
    { $match: { username: username}},
    <!-- Remove irrelevant information -->
    { $project: { tweets: 1, following: 1 }},
    <!-- Lookup to get tweets array from Following -->
    { $lookup: {
        from: "users",
        localField: "following",
        foreignField: "username",
        <!-- Keep only tweets array from following -->
        pipeline: [
            { $project: { tweets: 1 }}
        ],
        as: "data"
    }},
    <!-- Remove following from original user document now that we don't need it anymore -->
    { $project: { following: 0 }}
])

Return Value:

{
  _id: ObjectId("64f692b507474379dca9374d"),
  tweets: [
    ObjectId("64fc126f0fb11976be60988a"),
    ObjectId("64fc127d0fb11976be60988d"),
    ObjectId("64fc12890fb11976be609890")
  ],
  data: [
    {
      _id: ObjectId("64f982c2fc1555b92020cac7"),
      tweets: [
        ObjectId("64fc120d0fb11976be609882"),
        ObjectId("64fc121c0fb11976be609885")
      ]
    },
    {
      _id: ObjectId("64f98ec10fb11976be609865"),
      tweets: [
        ObjectId("64fc11c50fb11976be609879"),
        ObjectId("64fc11e50fb11976be60987c")
      ]
    }
  ]
}

This is close but I’m having trouble flattening the arrays especially since they’re nested. I tried something with $addToSet for the arrays inside data and concatenating them but I ended up having to chain 3 $unwind’s to get something desirable which feels hacky in the worst way.

{ 
        $group: { 
            _id: null,
            owner: {
                $addToSet: '$tweets'
            },
            following: {
                $addToSet: '$data.tweets'
            }
        }
    },
    { $project: {
        allTweets: {
            $concatArrays: ['$owner', '$following']
        }
    }},
    { $unwind: '$allTweets' },
    { $unwind: '$allTweets' },
    { $unwind: '$allTweets' },
    {
        $group: {
            _id: '$allTweets'
        }
    }

Which returned individual documents containing just each tweet with its ObejctId as the _id key:

{
  _id: ObjectId("64fc121c0fb11976be609885")
}
{
  _id: ObjectId("64fc121c0fb11976be609886")
}
{
  _id: ObjectId("64fc121c0fb11976be609887")
}
{
  _id: ObjectId("64fc121c0fb11976be609888")
}
{
  _id: ObjectId("64fc121c0fb11976be609889")
}

Could someone guide me towards the proper way of doing this? Thanks!

2

Someone else may provide more specific/deep help but I wanted to flag Asya’s comment on Efficient Structure for Social Media Feeds (fan-out on write)? - #3 by Asya_Kamsky as it may include some helpful links / a reference architecture related to what you’re doing