Indexes performing poorly on $in

Hi,

Data in someCollection : 100000 records
someCollection size : 7GB
Using atlas : M30 (8 GB RAM, 52 GB Storage) 3,000 IOPS, Encrypted, Auto-expand Storage

I am running a find query like following
db.someCollection.find({someId :{$in: [ <260 objectIds> ] }}).sort({_id:-1}).

Index applied : {someId:1,_id:-1}

Atlas is hosted in ca-central. I making the query from India. The query take more than 5 mins to run even though the index is applied. What is the possible issue ? Is the filter too large for the data ? Doesn’t mongo support large filter value? What are the possible solutions?

Primary node query execution plan:

{
  "queryPlanner": {
    "plannerVersion": 1,
    "namespace": "someDb.someCollection",
    "indexFilterSet": false,
    "parsedQuery": {
      "someId": {
        "$in": [
        ]
      }
    },
    "queryHash": "937709B1",
    "planCacheKey": "8A586950",
    "winningPlan": {
      "stage": "FETCH",
      "inputStage": {
        "stage": "SORT",
        "sortPattern": {
          "_id": -1
        },
        "memLimit": 104857600,
        "type": "default",
        "inputStage": {
          "stage": "IXSCAN",
          "keyPattern": {
            "someId": 1,
            "_id": -1
          },
          "indexName": "someId_1__id_-1",
          "isMultiKey": false,
          "multiKeyPaths": {
            "someId": [],
            "_id": []
          },
          "isUnique": false,
          "isSparse": false,
          "isPartial": false,
          "indexVersion": 2,
          "direction": "forward",
          "indexBounds": {
            "someId": [
              
            ],
            "_id": [
              "[MaxKey, MinKey]"
            ]
          }
        }
      }
    },
    "rejectedPlans": [
      {
        "stage": "SORT",
        "sortPattern": {
          "_id": -1
        },
        "memLimit": 104857600,
        "type": "simple",
        "inputStage": {
          "stage": "FETCH",
          "inputStage": {
            "stage": "IXSCAN",
            "keyPattern": {
              "someId": 1
            },
            "indexName": "someId_1",
            "isMultiKey": false,
            "multiKeyPaths": {
              "someId": []
            },
            "isUnique": false,
            "isSparse": false,
            "isPartial": false,
            "indexVersion": 2,
            "direction": "forward",
            "indexBounds": {
              "someId": [
              
              ]
            }
          }
        }
      },
      {
        "stage": "FETCH",
        "filter": {
          "someId": {
            "$in": [
             
            ]
          }
        },
        "inputStage": {
          "stage": "IXSCAN",
          "keyPattern": {
            "_id": 1
          },
          "indexName": "_id_",
          "isMultiKey": false,
          "multiKeyPaths": {
            "_id": []
          },
          "isUnique": true,
          "isSparse": false,
          "isPartial": false,
          "indexVersion": 2,
          "direction": "backward",
          "indexBounds": {
            "_id": [
              "[MaxKey, MinKey]"
            ]
          }
        }
      },
      {
        "stage": "FETCH",
        "filter": {
          "someId": {
            "$in": [
             
            ]
          }
        },
        "inputStage": {
          "stage": "IXSCAN",
          "keyPattern": {
            "_id": -1,
            "someId": 1
          },
          "indexName": "_id_-1_someId_1",
          "isMultiKey": false,
          "multiKeyPaths": {
            "_id": [],
            "someId": []
          },
          "isUnique": false,
          "isSparse": false,
          "isPartial": false,
          "indexVersion": 2,
          "direction": "forward",
          "indexBounds": {
            "_id": [
              "[MaxKey, MinKey]"
            ],
            "someId": [
              "[MinKey, MaxKey]"
            ]
          }
        }
      }
    ]
  },
  "serverInfo": {
    "host": "some.mongodb.net",
    "port": 27017,
    "version": "4.4.25",
    "gitVersion": ""
  },
  "ok": 1,
  "$clusterTime": {
    "clusterTime": "10/9/2023 20:37:07.000 (#1)",
    "signature": {
      "hash": "BinData(0,\"//VVuk=\")",
      "keyId": "someIddd"
    }
  },
  "operationTime": "10/9/2023 20:37:07.000 (#1)"
}

Hey @Kamaldeep_Kaur,

Welcome to the MongoDB Community!

Based on the shared image, it appears that the query execution time is around 378 ms. Please let me know if I’m missing something.

There is no hard-coded limit on the number of values in $in. But in general, very large $in lists with hundreds or thousands of items will negatively impact query performance.

However, if you need further assistance, please feel free to share a sample document (remember to redact any sensitive information) so that we can try to replicate it in our own environment.

Best regards,
Kushagra

“Based on the shared image, it appears that the query execution time is around 378 ms. Please let me know if I’m missing something.” - If I run db.someCollection.find({someId :{$in: [ <260 objectIds> ] }}).sort({_id:-1}).explain(“executionStats”) it shows 378 ms, but if run db.someCollection.find({someId :{$in: [ <260 objectIds> ] }}).sort({_id:-1}) it sometimes doesn’t even returns a result, if I check the profiler the docs returned are shown as null and time taken is around 15 min or so. Majority of cases nothing is returned further hampering our prod env.

Requested sample docs:
(Please note that some docs contain images and the size of an image is around 3.6MB each. There around 6 images , and also the signatures are encoded. Due to images in documents the size of doc is 15MB for around 9022 documents in the collection, otherwise the docs without the images have size around 90KB )

1. {
    "_id": "61ffffffffffffffffffffff",
    "isSomesome": false,
    "issometed": false,
    "someeeeeeeeBy": "Ksome Rrrr",
    "isSomeeee": true,
    "isSssssssssst": false,
    "dummyyyyId": "600000000000000000000000",
    "someeeId": "5ddddddddddddddddddddddd",
    "someId": "5drrrrrrrrrrrrrrrrrrrrrr",
    "asssssssssnkId": "5ffffffffffffffffffffff",
    "nameOfSigner": "Ksome Rrrr",
    "signerRole": "",
    "userAgent": "Hhhhhhhhh/3 CFfffffff/1111.0.1 Ssssin/22.6.0",
  "ipAddress": "111.111.11.111",
  "somedddId": "6555555555555555556666",
    "created": "2022-02-04T19:51:47.000Z",
    "updated": "2022-02-04T19:52:03.712Z",
    "noteAssessment": {
      "notes": "Forms",
      "isClearanceNote": false,
      "isPhysicalNote": false,
      "images": [],// 6 images size 3.6MB
      "date": "2022-02-04T19:49:00.000Z",
      "isNeuropsychNote": false,
      "injuryType": "",
      "attachments": [
        "613333333333333333333333/attachment_0.png",
        "61tttttttttttttttttttttt/attachment_1.png",
        "61fdyyyyyy55555555555555/attachment_2.png",
        "61fdwwwwwwwwwwww33333333/attachment_3.png",
        "61f666666666666666666666/attachment_4.png"
      ]
    }
  }

2. 
{
  "_id": "xx",
  "isDummyyy": false,
  "someeeeeeeeBy": "Ttt Ccccccc",
  "isSomeeee": false,
  "dummyyyyId": "666666666666666666666666", //Objectid
  "someeeId": "62222222222222222222222",//Objectid
  "someId": "61111111111111111111111",//Objectid
  "someeeeSignature": "encoded",
  "nameOfSigner": "Ttt Ccccccc",
  "signerRole": "Hhhh Ttttttttt/Ttttttt",
  "someeeeeeeType": "sssssSsssssss",
  "someersion": "7.2.0",
  "userAgent": "Hhhhhhhhh/3 CFfffffff/1111.0.1 Ssssin/22.6.0",
  "ipAddress": "111.111.11.111",
  "somedddId": "64444444444666666666666", //objectId
  "created": "2023-08-18T16:56:36.030Z",
  "updated": "2023-08-18T17:06:43.524Z",
  "basomeeYest": {
    "sinssssssssssssFfffDssssse": 1,
    "doSSSSSeeeeeeeeeeeeeeeeeeeee": 1,
    "tasomeStsomeOnFirmSurface": 1,
    "wasPerformed": true,
    "nonDominantFoot": "left",
    "testingSurface": "hardFloor",
    "footwear": "shoes"
  },
  "cognitiveAssessment": {
    "whatMonth": true,
    "whatDate": true,
    "whatDay": true,
    "whatYear": true,
    "whatTime": true,
    "trialFirst": 10,
    "trialSecond": 10,
    "trialThird": 10,
    "numbersFirst": true,
    "numbersSecond": true,
    "numbersThird": true,
    "numbersFourth": true,
    "valueMemoryReverseMonthTime": 7.292173981666565,
    "valueMemorySecondFirst": 1,
    "valueMemorySecondSecond": 1,
    "valueMemorySecondThird": 1,
    "valueMemorySecondFourth": 1,
    "valueMemorySecondFive": 1,
    "valueMemorySecondSix": 1,
    "valueMemorySecondSeven": 1,
    "valueMemorySecondEight": 1,
    "valueMemorySecondNine": 1,
    "valueMemorySecondTen": 1,
    "digits": "4-9-3 6-2-9, 3-8-1-4 3-2-7-9, 6-2-9-7-1 1-5-2-8-6, 7-1-8-4-6-2 5-3-9-1-4-8",
    "wordNames": "Baby, Monkey, Perfume, Sunset, Iron, Elbow, Apple, Carpet, Saddle, Bubble",
    "firstTrialSelectedValues": "1,1,1,1,1,1,1,1,1,1",
    "secondTrialSelectedValues": "1,1,1,1,1,1,1,1,1,1",
    "thirdTrialSelectedValues": "1,1,1,1,1,1,1,1,1,1",
    "immediateMemoryCompletedTime": "2023-08-18T16:58:25.189Z",
    "delayedRecallStartedTime": "2023-08-18T17:06:29.344Z",
    "wasPerformed": true,
    "immediateMemoryTrialsAudio": [
      "trialFirst",
      "trialThird"
    ],
    "digitsAudio": [
      "4-9-3 6-2-9",
      "3-8-1-4 3-2-7-9",
      "6-2-9-7-1 1-5-2-8-6",
      "7-1-8-4-6-2 5-3-9-1-4-8"
    ],
    "language": "eng",
    "valueReverseMonthsCorrect": ""
  },
  "symptomEvaluation": {
    "headache": 0,
    "pressureInHead": 0,
    "neckPain": 0,
    "nauseaOrVomiting": 0,
    "dizziness": 0,
    "blurredVision": 0,
    "balanceProblems": 0,
    "sensivityToLight": 0,
    "sensivityToNoice": 0,
    "feelingSlowedDown": 0,
    "feelingLikeInAFog": 0,
    "dontFeelRight": 0,
    "difficultyConcentraiting": 0,
    "difficultyRemembering": 0,
    "fatigueOrLowEnergy": 0,
    "confusion": 0,
    "drawsiness": 0,
    "moreEmotional": 0,
    "irritability": 1,
    "sadness": 0,
    "nervousOrAnxious": 0,
    "worseActivity": 0,
    "worseMentalActivity": 0,
    "feelingNotNormal": 97,
    "wasPerformed": true,
    "assssssSignature": "encoded",
    "language": "english"
  },
  "someIId": "64dddddddddddddddddddddd", //objectid
  "tandemGait": {
    "trial1": 6.467267036437988,
    "trial2": 6.117172002792358,
    "trial3": 6.384270071983337,
    "wasPerformed": true,
    "ableToPerformTandemGait": true,
    "ableToPerformTandemGaitNotes": "Sprained ankle "
  },
  "dualTaskGait": {
    "firstTrialTime": 23.499022006988525,
    "practiceTrialValues": [
      {
        "expectedAnswer": 93,
        "givenAnswer": 93
      },
      {
        "expectedAnswer": 86,
        "givenAnswer": 86
      },
      {
        "expectedAnswer": 79,
        "givenAnswer": 79
      },
      {
        "expectedAnswer": 72,
        "givenAnswer": 72
      },
      {
        "expectedAnswer": 65,
        "givenAnswer": 65
      },
      {
        "expectedAnswer": 58,
        "givenAnswer": 58
      },
      {
        "expectedAnswer": 51,
        "givenAnswer": 51
      },
      {
        "expectedAnswer": 44,
        "givenAnswer": 44
      },
      {
        "expectedAnswer": 37
      }
    ],
    "secondTrialValues": [
      {
        "expectedAnswer": 90,
        "givenAnswer": 83
      },
      {
        "expectedAnswer": 76,
        "givenAnswer": 76
      },
      {
        "expectedAnswer": 69,
        "givenAnswer": 69
      },
      {
        "expectedAnswer": 62,
        "givenAnswer": 62
      },
      {
        "expectedAnswer": 55,
        "givenAnswer": 55
      },
      {
        "expectedAnswer": 48,
        "givenAnswer": 48
      },
      {
        "expectedAnswer": 41,
        "givenAnswer": 41
      },
      {
        "expectedAnswer": 34
      }
    ],
    "wasPerformed": true,
    "secondTrialStartingValue": 90,
    "thirdTrialValues": [
      {
        "expectedAnswer": 98,
        "givenAnswer": 98
      },
      {
        "expectedAnswer": 91,
        "givenAnswer": 92
      },
      {
        "expectedAnswer": 85,
        "givenAnswer": 85
      },
      {
        "expectedAnswer": 78,
        "givenAnswer": 78
      },
      {
        "expectedAnswer": 71,
        "givenAnswer": 71
      },
      {
        "expectedAnswer": 64,
        "givenAnswer": 64
      },
      {
        "expectedAnswer": 57,
        "givenAnswer": 57
      },
      {
        "expectedAnswer": 50
      }
    ],
    "secondTrialTime": 20.729416012763977,
    "thirdTrialStartingValue": 98,
    "firstTrialStartingValue": 88,
    "thirdTrialTime": 17.931267976760864,
    "practiceTime": 18.905928015708923,
    "firstTrialValues": [
      {
        "expectedAnswer": 88,
        "givenAnswer": 88
      },
      {
        "expectedAnswer": 81,
        "givenAnswer": 81
      },
      {
        "expectedAnswer": 74,
        "givenAnswer": 75
      },
      {
        "expectedAnswer": 68,
        "givenAnswer": 68
      },
      {
        "expectedAnswer": 61,
        "givenAnswer": 61
      },
      {
        "expectedAnswer": 54,
        "givenAnswer": 52
      },
      {
        "expectedAnswer": 45,
        "givenAnswer": 45
      },
      {
        "expectedAnswer": 38
      }
    ],
    "subtractValue": 7
  }
}

@Kushagra_Kesav There’s another case I would like to discuss for the same collection.
I have a query like following
db.someCollection.find({someKey: {$exists: false},someId:{$in:[<260 objectids>] }}).sort({_id:1})

If I apply index on > {someKey:1, someId:1, _id:1} and run the above query with 180 objectids the index is applied. But if I run the above query as is on 260 objectids, the it does the IXSCAN but on _id. Not sure what is the issue ?
Seems like fields with $exists : false do not accept indexes
The other finding is that if I only apply index on { someId:1, _id:1} the query still uses IXSCAN on _id whereas i believe it should use IXSCAN on { someId:1, _id:1}.
Please help