MongoError: document constructed by $facet is 104860008 bytes, which exceeds the limit of 104857600 bytes

Working with Data
1

Hi All,

I am getting memory size limit error while running multiple sub-pipelines in $facet. Can someone help me on this issue.
Scenario: I have a crown job which runs once a day. I want to execute a pipeline in it against a collection with millions of documents.

[ 
     {
          $facet: {
                  query1: [pipeline_1],
                  query2: [pipeline_2],
                  query3: [pipeline_3]
                  ...
                  query_n: [pipeline_n]
          },
     },
     {
        $merge:{ into: some_collection}
     }
]

I tried allowDiskUse=true, But still giving same error.

What can be the work around on this. Please help.

2

Hi @Pradip_Kumar,

How did you specified the allowDiskUse: true?

AFAIK, this stage should allow it to be used.

Thanks
Pavel

3

I used below syntax from CLI:

db.collection_name.aggregate( [pipeline], {allowDiskUse: true}}

and from pymango i am using below syntax :

db_client.aggregate(
db_name, collection_name, pipeline, allow_disk_use=True)

It is not working in both cases.

4

@Pradip_Kumar,

According to PyMongo the parameter name is still called allowDiskUse:
https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.aggregate

What I suspect is that the error is not rrelated to the 100MB per stage limit, but maybe to the output of $facet object that we get from your agg…

This object is not suppose to cross 16MB actually.

1 Like
5

I am running into the this same exact issue. My error is:

MongoError: document constructed by $facet is 104857822 bytes, which exceeds the limit of 104857600 bytes

I understand the 100M state limit, but shouldn’t allowDiskUse:true address that? I am using mongoose and am passing that parameter like so:
Collection.aggreagate([…]).allowDiskUse(true).then(result => {…

You seem to suggest that the issue has to do with the 16MB doc limit. Why, then, does is mention the “the limit of 104857600 bytes” which is about 100MB? I don’t see it complaining about 16 MB anywhere.

My aggregation looks like this:

        [
            {
                "$facet": {
                    "emails": [
                        {
                            "$project": {
                                _id: false,
                                "k": "$email",
                                "v": "1"
                            }
                        }
                    ]
                }
            },
            {
                "$project": {
                    "emails": {
                        $arrayToObject: "$emails"
                    }
                }
            }
        ]