How to use aggregation for large collection?

Working with Data
6

@Asya_Kamsky thanks for stepping in. The reason why I stumbled on Eden’s post is that I had this problem myself, and I was looking for the best way to solve it.

Let me describe my use case, our web app handles stock market transactions (aka account “activities”), and we use a Bucket Pattern, because many of our users have several 20k-50k transactions/activities in their account (i.e. several times the 16MB limit). Our use case is pretty much exactly the example described in these two articles by Justin LaBreck.

I was getting BSON size limit error messages from the following query when querying users with many activityBuckets. I added the project: {"activities": 0} stage and it solved my problem. The query returns all of the user’s activityBuckets, but without the actual activity data (ie. only the activityBucket high level data).

Would you have recommended a different solution?

//-----------------------------------------------------------------------------------------------------
// get user and all its activityBuckets(without actual activities otherwise would bust 16MB)
//-----------------------------------------------------------------------------------------------------
db.users.aggregate( [
    { $match: { 'email': 'johndoe@featherfinance.com' } }, 
    { $lookup: {
        from: "activitybuckets",
        let: { users_id: "$_id"},
        pipeline: [ 
            { $project: {"activities": 0} },
            {
                $match: {
                    $expr: { 
                        $and: [
                            { $eq: [ '$$users_id', "$user" ] },
                    }
                }
            }
        ],
        as: "activities"
    } },
] );