FETCH Performance degradation after upgrade from v7.0 to v8.0

Working with Data
, ,
1

I recently ran performance benchmarks with MongoDB v7.0.5 and MongoDB v8.0.1 to see potential gains for some clients. Instead of making gains, I suffered serious performance degradation. Operation read/write latency histograms look much worse, and I have over 103k “Slow query” logs (“inserts”) in v8 while in the old version, I had only 6k of them when running the same operations.

I profiled it with Studio 3T to see some problematic queries, and I saw a specific query that needs 1100 seconds to finish instead of 30 seconds.

db.getSiblingDB("DataDb").getCollection("67a27384313237fc7f2fa1dc.Records").aggregate(
    [
        {
            "$match" : {
                "DeliveryId" : ObjectId("60e2be5d0fa1622e101ac644")
            }
        }
    ], 
    {
        "allowDiskUse" : true
    }
)

according to execution stats, I have the biggest performance drop on the ‘FETCH’ stage:
MongoDB v8.0.1:

  executionStats: {
    executionSuccess: true,
    nReturned: 3109409,
    executionTimeMillis: 1403432,
    totalKeysExamined: 3109409,
    totalDocsExamined: 3109409,
    executionStages: {
      isCached: false,
      stage: 'FETCH',
      nReturned: 3109409,
      executionTimeMillisEstimate: 1328062,
      works: 3109410,
      advanced: 3109409,
      needTime: 0,
      needYield: 0,
      saveState: 45999,
      restoreState: 45999,
      isEOF: 1,
      docsExamined: 3109409,
      alreadyHasObj: 0,
      inputStage: {
        stage: 'IXSCAN',
        nReturned: 3109409,
        executionTimeMillisEstimate: 2644,
        works: 3109410,
        advanced: 3109409,
        needTime: 0,
        needYield: 0,
        saveState: 45999,
        restoreState: 45999,
        isEOF: 1,
        keyPattern: { DeliveryId: 1, RecordId: 1, DataSource: 1, Revision: 1 },
        indexName: 'DI_RI_DS_R',
        isMultiKey: false,
        multiKeyPaths: { DeliveryId: [], RecordId: [], DataSource: [], Revision: [] },
        isUnique: true,
        isSparse: false,
        isPartial: false,
        indexVersion: 2,
        direction: 'forward',
        indexBounds: {
          DeliveryId: [
            "[ObjectId('60e2be5d0fa1622e101ac644'), ObjectId('60e2be5d0fa1622e101ac644')]"
          ],
          RecordId: [ '[MinKey, MaxKey]' ],
          DataSource: [ '[MinKey, MaxKey]' ],
          Revision: [ '[MinKey, MaxKey]' ]
        },
        keysExamined: 3109409,
        seeks: 1,
        dupsTested: 0,
        dupsDropped: 0
      }
    }

vs MongoDB version 7.0.5:

executionStats: {
executionSuccess: true,
nReturned: 3109409,
executionTimeMillis: 30655,
totalKeysExamined: 3109409,
totalDocsExamined: 3109409,
executionStages: {
stage: ‘FETCH’,
nReturned: 3109409,
executionTimeMillisEstimate: 15533,
works: 3109410,
advanced: 3109409,
needTime: 0,
needYield: 0,
saveState: 3187,
restoreState: 3187,
isEOF: 1,
docsExamined: 3109409,
alreadyHasObj: 0,
inputStage: {
stage: ‘IXSCAN’,
nReturned: 3109409,
executionTimeMillisEstimate: 1124,
works: 3109410,
advanced: 3109409,
needTime: 0,
needYield: 0,
saveState: 3187,
restoreState: 3187,
isEOF: 1,
keyPattern: { DeliveryId: 1, RecordId: 1, DataSource: 1, Revision: 1 },
indexName: ‘DI_RI_DS_R’,
isMultiKey: false,
multiKeyPaths: { DeliveryId: , RecordId: , DataSource: , Revision:  },
isUnique: true,
isSparse: false,
isPartial: false,
indexVersion: 2,
direction: ‘forward’,
indexBounds: {
DeliveryId: [
“[ObjectId(‘60e2be5d0fa1622e101ac644’), ObjectId(‘60e2be5d0fa1622e101ac644’)]”
],
RecordId: [ ‘[MinKey, MaxKey]’ ],
DataSource: [ ‘[MinKey, MaxKey]’ ],
Revision: [ ‘[MinKey, MaxKey]’ ]
},
keysExamined: 3109409,
seeks: 1,
dupsTested: 0,
dupsDropped: 0
}
}

HW remained the same, I didn’t add any additional parameters to mongod.cfg file, in the getParameter admin Command I found only these differences (besides parameters that are present only in v7 or v8):

parameter name MognoDB v8 MongoDB v7
allowUnsafeUntimestampedWrites false true
internalQueryCardinalityEstimatorMode heuristic sampling
tenantMigrationDisableX509Auth true false
maxSizeOfBatchedInsertsForRenameAcrossDatabasesBytes 4194304 16776216
internalQueryExecYieldIterations -1 1000

Additional information about the DB:
Documents: ~3M
Total size: 41.8GB
Storage Size: 12.3GB
Indexes: 10
Total Index size: 492MB

What else can I do to investigate the issue? Are there any specific parameters that can help with fetching records?