Mongodb Aggregate / $facet query is too slow. its taking 20sec

Hi,

We have implemented the aggregation in the app. But the aggregation query is taking 20 sec to load.

We have the $match condition $facet stage.

Below is the sample query:

db.used_vehicles.explain(“executionStats”).aggregate([{“$match”:{“status”:“ACTIVE”,“media_exist”:1,“vehicle_type_id”:1,“listing_status”:“APPROVED”}},{“$facet”:{“categoryByState”:[{“$match”:{“used_vehicle_spec.state.state_name”:{“$ne”:null}}},{“$group”:{“_id”:“$used_vehicle_spec.state.state_id”,“state_identifier”:{“$first”:“$used_vehicle_spec.state.state_identifier”},“state_name”:{“$first”:“$used_vehicle_spec.state.state_name”},“state_count”:{“$sum”:1}}},{“$project”:{“_id”:0,“state_id”:“$_id”,“state_identifier”:“$state_identifier”,“state_name”:“$state_name”,“state_count”:“$state_count”}},{“$sort”:{“state_count”:-1,“state_name”:1}}],“categoryByCity”:[{“$match”:{“used_vehicle_spec.city.city_name”:{“$ne”:“”}}},{“$group”:{“_id”:“$used_vehicle_spec.city.city_id”,“city_identifier”:{“$first”:“$used_vehicle_spec.city.city_identifier”},“city_name”:{“$first”:“$used_vehicle_spec.city.city_name”},“state_id”:{“$first”:“$used_vehicle_spec.city.state_id”},“city_count”:{“$sum”:1}}},{“$project”:{“_id”:0,“city_id”:“$_id”,“city_identifier”:“$city_identifier”,“city_name”:“$city_name”,“state_id”:“$state_id”,“city_count”:“$city_count”}},{“$sort”:{“city_count”:-1,“city_name”:1}}],“categoryByBrand”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“brand.brand_name”:{“$ne”:null}}},{“$group”:{“_id”:“$brand.brand_id”,“brand_name”:{“$first”:“$brand.brand_name”},“count”:{“$sum”:1}}},{“$project”:{“_id”:0,“brand_id”:“$_id”,“brand_name”:“$brand_name”,“brand_count”:“$count”}},{“$sort”:{“brand_count”:-1,“brand_name”:1}}],“categoryByModel”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“model.model_name”:{“$ne”:null}}},{“$group”:{“_id”:“$model.model_id”,“brand_id”:{“$first”:“$brand.brand_id”},“model_name”:{“$first”:“$model.model_name”},“count”:{“$sum”:1}}},{“$project”:{“_id”:0,“model_id”:“$_id”,“model_name”:“$model_name”,“brand_id”:“$brand_id”,“model_count”:“$count”}},{“$sort”:{“model_count”:-1,“model_name”:1}}],“categoryByBody”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“variant.body_type.shape_name”:{“$ne”:null}}},{“$group”:{“_id”:“$variant.body_type.shape_id”,“shape_name”:{“$first”:“$variant.body_type.shape_name”},“body_count”:{“$sum”:1}}},{“$project”:{“_id”:0,“body_type_id”:“$_id”,“body_type_name”:“$shape_name”,“body_count”:“$body_count”}},{“$sort”:{“body_count”:-1,“body_type_name”:1}}],“categoryByFuel”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“variant.fuel_type.fuel_name”:{“$ne”:null}}},{“$group”:{“_id”:“$variant.fuel_type.fuel_id”,“fuel_name”:{“$first”:“$variant.fuel_type.fuel_name”},“fuel_count”:{“$sum”:1}}},{“$project”:{“_id”:0,“fuel_type_id”:“$_id”,“fuel_type_name”:“$fuel_name”,“fuel_type_count”:“$fuel_count”}},{“$sort”:{“fuel_type_count”:-1,“fuel_type_name”:1}}],“categoryBytransmission”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“variant.transmission_type”:{“$ne”:“”}}},{“$group”:{“_id”:“$variant.transmission_type”,“transmission_count”:{“$sum”:1}}},{“$project”:{“_id”:0,“transmission_type”:“$_id”,“transmission_count”:“$transmission_count”}},{“$sort”:{“transmission_count”:-1,“transmission_type”:1}}],“categoryBySeller”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“certified_seller_data.seller_type”:{“$ne”:null}}},{“$group”:{“_id”:“$certified_seller_data.seller_type”,“seller_count”:{“$sum”:1}}},{“$project”:{“_id”:0,“seller_type”:“$_id”,“seller_count”:“$seller_count”}},{“$sort”:{“seller_count”:-1,“seller_type”:1}}],“categoryByOwner”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“used_vehicle_spec.no_of_owners”:{“$ne”:null}}},{“$group”:{“_id”:“$used_vehicle_spec.no_of_owners”,“owner_count”:{“$sum”:1}}},{“$project”:{“_id”:0,“no_of_owners”:“$_id”,“owner_count”:“$owner_count”}},{“$sort”:{“owner_count”:-1,“no_of_owners”:1}}],“categoryByRegister”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“used_vehicle_spec.register_type”:{“$ne”:“”}}},{“$group”:{“_id”:“$used_vehicle_spec.register_type”,“register_count”:{“$sum”:1}}},{“$project”:{“_id”:0,“register_type”:“$_id”,“register_count”:“$register_count”}},{“$sort”:{“register_count”:-1,“register_type”:1}}],“categoryByCertified”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“certified”:{“$ne”:null}}},{“$group”:{“_id”:“$certified”,“certified_count”:{“$sum”:1}}},{“$project”:{“_id”:0,“is_certified”:“$_id”,“certified_count”:“$certified_count”}},{“$sort”:{“certified_count”:-1}}],“categoryByPrice”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“price”:{“$ne”:“”}}},{“$bucket”:{“groupBy”:“$price”,“boundaries”:[0,500000,1000000,2500000,5000000,7500000,10000000],“default”:10000000,“output”:{“count”:{“$sum”:1}}}},{“$addFields”:{“upperBound”:{“$arrayElemAt”:[[0,500000,1000000,2500000,5000000,7500000,10000000],{“$sum”:[{“$indexOfArray”:[[0,500000,1000000,2500000,5000000,7500000,10000000],“$_id”]},1]}]}}}],“categoryByYear”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“used_vehicle_spec.mfg_year”:{“$ne”:null}}},{“$bucket”:{“groupBy”:“$used_vehicle_spec.mfg_year”,“boundaries”:[0,2011,2014,2017,2019,2022],“default”:2023,“output”:{“count”:{“$sum”:1}}}},{“$addFields”:{“upperBound”:{“$arrayElemAt”:[[0,2011,2014,2017,2019,2022],{“$sum”:[{“$indexOfArray”:[[0,2011,2014,2017,2019,2022],“$_id”]},1]}]}}}],“categoryByKms”:[{“$match”:{“used_vehicle_spec.city.city_id”:{“$in”:[349]},“used_vehicle_spec.kms_driven”:{“$ne”:null}}},{“$bucket”:{“groupBy”:“$used_vehicle_spec.kms_driven”,“boundaries”:[0,30000,60000,100000],“default”:100000,“output”:{“count”:{“$sum”:1}}}},{“$addFields”:{“upperBound”:{“$arrayElemAt”:[[0,30000,60000,100000],{“$sum”:[{“$indexOfArray”:[[0,30000,60000,100000],“$_id”]},1]}]}}}]}}])

That’s pretty unreadable, can you paste it in as a code fragment with formatting, use the “Code” formatting on the editbox toolbar.

1 Like

HI john
Thanks for the update. Here you can find the formatted query code. Please help us to solve this problem.

used_vehicles.aggregate([
  {
    "$match": {
      "status": "ACTIVE",
      "media_exist": 1,
      "vehicle_type_id": 1,
      "listing_status": "APPROVED"
    }
  },
  {
    "$facet": {
      "categoryByState": [
        { "$match": { "used_vehicle_spec.state.state_name": { "$ne": null } } },
        {
          "$group": {
            "_id": "$used_vehicle_spec.state.state_id",
            "state_identifier": {
              "$first": "$used_vehicle_spec.state.state_identifier"
            },
            "state_name": { "$first": "$used_vehicle_spec.state.state_name" },
            "state_count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "state_id": "$_id",
            "state_identifier": "$state_identifier",
            "state_name": "$state_name",
            "state_count": "$state_count"
          }
        },
        { "$sort": { "state_count": -1, "state_name": 1 } }
      ],
      "categoryByCity": [
        { "$match": { "used_vehicle_spec.city.city_name": { "$ne": "" } } },
        {
          "$group": {
            "_id": "$used_vehicle_spec.city.city_id",
            "city_identifier": {
              "$first": "$used_vehicle_spec.city.city_identifier"
            },
            "city_name": { "$first": "$used_vehicle_spec.city.city_name" },
            "state_id": { "$first": "$used_vehicle_spec.city.state_id" },
            "city_count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "city_id": "$_id",
            "city_identifier": "$city_identifier",
            "city_name": "$city_name",
            "state_id": "$state_id",
            "city_count": "$city_count"
          }
        },
        { "$sort": { "city_count": -1, "city_name": 1 } }
      ],
      "categoryByBrand": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "brand.brand_name": { "$ne": null }
          }
        },
        {
          "$group": {
            "_id": "$brand.brand_id",
            "brand_name": { "$first": "$brand.brand_name" },
            "count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "brand_id": "$_id",
            "brand_name": "$brand_name",
            "brand_count": "$count"
          }
        },
        { "$sort": { "brand_count": -1, "brand_name": 1 } }
      ],
      "categoryByModel": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "model.model_name": { "$ne": null }
          }
        },
        {
          "$group": {
            "_id": "$model.model_id",
            "brand_id": { "$first": "$brand.brand_id" },
            "model_name": { "$first": "$model.model_name" },
            "count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "model_id": "$_id",
            "model_name": "$model_name",
            "brand_id": "$brand_id",
            "model_count": "$count"
          }
        },
        { "$sort": { "model_count": -1, "model_name": 1 } }
      ],
      "categoryByBody": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "variant.body_type.shape_name": { "$ne": null }
          }
        },
        {
          "$group": {
            "_id": "$variant.body_type.shape_id",
            "shape_name": { "$first": "$variant.body_type.shape_name" },
            "body_count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "body_type_id": "$_id",
            "body_type_name": "$shape_name",
            "body_count": "$body_count"
          }
        },
        { "$sort": { "body_count": -1, "body_type_name": 1 } }
      ],
      "categoryByFuel": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "variant.fuel_type.fuel_name": { "$ne": null }
          }
        },
        {
          "$group": {
            "_id": "$variant.fuel_type.fuel_id",
            "fuel_name": { "$first": "$variant.fuel_type.fuel_name" },
            "fuel_count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "fuel_type_id": "$_id",
            "fuel_type_name": "$fuel_name",
            "fuel_type_count": "$fuel_count"
          }
        },
        { "$sort": { "fuel_type_count": -1, "fuel_type_name": 1 } }
      ],
      "category transmission": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "variant.transmission_type": { "$ne": "" }
          }
        },
        {
          "$group": {
            "_id": "$variant.transmission_type",
            "transmission_count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "transmission_type": "$_id",
            "transmission_count": "$transmission_count"
          }
        },
        { "$sort": { "transmission_count": -1, "transmission_type": 1 } }
      ],
      "categoryBySeller": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "certified_seller_data.seller_type": { "$ne": null }
          }
        },
        {
          "$group": {
            "_id": "$certified_seller_data.seller_type",
            "seller_count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "seller_type": "$_id",
            "seller_count": "$seller_count"
          }
        },
        { "$sort": { "seller_count": -1, "seller_type": 1 } }
      ],
      "categoryByOwner": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "used_vehicle_spec.no_of_owners": { "$ne": null }
          }
        },
        {
          "$group": {
            "_id": "$used_vehicle_spec.no_of_owners",
            "owner_count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "no_of_owners": "$_id",
            "owner_count": "$owner_count"
          }
        },
        { "$sort": { "owner_count": -1, "no_of_owners": 1 } }
      ],
      "categoryByRegister": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "used_vehicle_spec.register_type": { "$ne": "" }
          }
        },
        {
          "$group": {
            "_id": "$used_vehicle_spec.register_type",
            "register_count": { "$sum": 1 }
          }
        },
        {
          "$project": {
            "_id": 0,
            "register_type": "$_id",
            "register_count": "$register_count"
          }
        },
        { "$sort": { "register_count": -1, "register_type": 1 } }
      ],
      "categoryByCertified": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "certified": { "$ne": null }
          }
        },
        { "$group": { "_id": "$certified", "certified_count": { "$sum": 1 } } },
        {
          "$project": {
            "_id": 0,
            "is_certified": "$_id",
            "certified_count": "$certified_count"
          }
        },
        { "$sort": { "certified_count": -1 } }
      ],
      "categoryByPrice": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "price": { "$ne": "" }
          }
        },
        {
          "$bucket": {
            "groupBy": "$price",
            "boundaries": [
              0, 500000, 1000000, 2500000, 5000000, 7500000, 10000000
            ],
            "default": 10000000,
            "output": { "count": { "$sum": 1 } }
          }
        },
        {
          "$addFields": {
            "upperBound": {
              "$arrayElemAt": [
                [0, 500000, 1000000, 2500000, 5000000, 7500000, 10000000],
                {
                  "$sum": [
                    {
                      "$indexOfArray": [
                        [
                          0, 500000, 1000000, 2500000, 5000000, 7500000,
                          10000000
                        ],
                        "$_id"
                      ]
                    },
                    1
                  ]
                }
              ]
            }
          }
        }
      ],
      "categoryByYear": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "used_vehicle_spec.mfg_year": { "$ne": null }
          }
        },
        {
          "$bucket": {
            "groupBy": "$used_vehicle_spec.mfg_year",
            "boundaries": [0, 2011, 2014, 2017, 2019, 2022],
            "default": 2023,
            "output": { "count": { "$sum": 1 } }
          }
        },
        {
          "$addFields": {
            "upperBound": {
              "$arrayElemAt": [
                [0, 2011, 2014, 2017, 2019, 2022],
                {
                  "$sum": [
                    {
                      "$indexOfArray": [
                        [0, 2011, 2014, 2017, 2019, 2022],
                        "$_id"
                      ]
                    },
                    1
                  ]
                }
              ]
            }
          }
        }
      ],
      "categoryByKms": [
        {
          "$match": {
            "used_vehicle_spec.city.city_id": { "$in": [349] },
            "used_vehicle_spec.kms_driven": { "$ne": null }
          }
        },
        {
          "$bucket": {
            "groupBy": "$used_vehicle_spec.kms_driven",
            "boundaries": [0, 30000, 60000, 100000],
            "default": 100000,
            "output": { "count": { "$sum": 1 } }
          }
        },
        {
          "$addFields": {
            "upperBound": {
              "$arrayElemAt": [
                [0, 30000, 60000, 100000],
                {
                  "$sum": [
                    { "$indexOfArray": [[0, 30000, 60000, 100000], "$_id"] },
                    1
                  ]
                }
              ]
            }
          }
        }
      ]
    }
  }
])

Thats a monster of a query, what have you done so far to analyse performance? Have you looked at an explain of the query?
How much data is in your collections and what indexes do you have, what does a document look like?

I should probably add, what are you trying to do and how often does this run, is it a key query that’s run? How often does the data update that would affect the output of it?

Hi John.

Yup.its primary query. the application related used vehicle ecommerce.

So we have filter with different components vehicles and need to show the count of vehicles by default.

If the user changes the filter combination, need to the count depends on that.

So we used facet.

Moreover, we have different combinations in this filter.
So possible combination also high. We can able to create 64 index …

How we can handle these scenarios to handle the compound index also.

Thanks in advance.

If this is a prime query and it takes this kind of query to get the data that’s used often in the right format then it kind of indicates that you may want to look at the schema.

We still don’t know what a document looks like, so I’ll not comment further on that but lots of the queries above look like they are filtering on the same criteria WITHIN the facet, this means the server has to repeat work for every facet, you also need to watch out for index use within a facet. Example is the filtering on city_id, perhaps split those up into a new pipeline so that you can share the filtering on all the items that use that criteria.

I’d start with one facet and try and look at how that’s performing and then work out from that. Don’t get hung up on trying to do everything in one pipeline stage if it’ll cripple your performance.

1 Like

Hi John.

Thanks for the update.

We tried with each facet in different aggregate pipeline and added index for match condition. Now that latency got reduced.

But still I have an doubt on index creation.

If we have multiple combination for filter,how do we create index for all combinations

Ex. Application have 15 type filter in page. So user can do the different combinations of filter in application.

How do we handle all the scenarios to create index

That’s a tricky one, perhaps someone who’s had to do this can comment, but as you say if you have lots of possible filters then indexes could quickly spiral out of control.
If each index is sufficiently limiting, then you could just have one on each field and rely on a non-indexed filter to kick in after that

In our application, we have lots of fields the user can filter on as well but the users mainly use a field that is sufficiently unique to reduce the amount of data to filter without the index to a handful, so it works out well.

Looks like the Attribute Pattern might be the way to go as 1 index might be able to accommodate the multiple combination.