Recently we have encountered query termination due to folowing error
2022-09-08T11:41:21.450-0400 W QUERY [conn590881] GetMore command executor error: FAILURE, status: CursorKilled: operation was interrupted, stats: { stage: "CACHED_PLAN" nReturned: 3327, executionTimeMillisEstimate: 57138, works: 7110357, advanced: 3327, needTime: 7107030, needYield: 0, saveState: 55897, restoreState: 55896, isEOF: 0, inputStage: { stage: "LIMIT", nReturned: 3327, executionTimeMillisEstimate: 55890, works: 7144006, advanced: 3327, needTime: 7140679, needYield: 0, saveState: 55897, restoreState: 55896, isEOF: 0, limit Amount: 2147483647, inputStage: stage: "PROJECTION DEFAULT", nReturned: 3327, executionTimeMillisEstimate: 55526, works: 7144006, advanced: 3327, needTime: 7140679, needYield: 0, saveState: 55897, restoreState: 55896, isEOF: 0, transformBy: { id: 0, XrefProperties: 0, XrefValues: 0, $sortKey: $meta: "sortKey" } }, inputStage: stage: "SORT_KEY_GENERATOR", nReturned: 3327, executionTimeMillisEstimate: 55219, works: 7144006, advanced: 3327, needTime: 7140679, needYield: 0, saveState: 55897, restoreState: 55896, isEOF: 0, inputStage: { stage: "FETCH", filter: $and: [ CalendarXrefValue: $in: [ "BkB", "BkS", "BmB", "BmS", "BOB" ]} }, CalendarDayName: { $not: $eq: "Weekend" } } }, { HolidayCode: { $not: { $eq: "712" } } } ] }, nReturned: 3327, executionTimeMillisEstimate: 54792, works: 7144005, advanced: 3327, needTime: 7140678, needYield: 0, saveState: 55897, restoreState: 55896, isEOF: 0, docsExamined: 7144005, alreadyHasObj: 0, inputStage: stage: "IXSCAN", nReturned: 7144005, executionTimeMillisEstimate: 10703, works: 7144005, advanced: 7144005, needTime: 0, needYield: 0, saveState: 55897, restoreState: 55896, is EOF: 0, keyPattern: { CalendarDate: 1.0, HolidayCode: 1.0 ), indexName: "CalendarDate_1_HolidayCode 1", isMultikey: false, multikeyPaths: { CalendarDate: [], HolidayCode: [] }, isUnique: false, isSparse: false, isPartial: false, indexVersion: 2, direction: "forward", indexBounds: CalendarDate: [[MinKey, MaxKey]" 1, HolidayCode: [ "[MinKey, Maxkey]" ] }, keysExamined: 7144005, seeks: 1, dups Tested: 0, dups Dropped: 0 } } } } } }
We run 4.2 sharded cluster under rhel7. There was no resource shortage on the server.
What could be root cause of this error and how can we prevent it from happening again?