Hi,
I am using mongorestore in a Kubernetes job. Sometimes, the job fails randomly especially when the db I’m restoring is a big one. Restarting the exact same k8s job sometimes works if the db I’m restoring is small.
It seems to be a shared resource issue but I am not sure what could it be. I tried to increase the resources for the job but it didn’t work it’s still failing with same error.
It fails with following error:
2022-06-12T15:01:01.937+0000 building indexes up to 4 collections in parallel
2022-06-12T15:01:01.937+0000 starting index build routine with id=3
2022-06-12T15:01:01.937+0000 starting index build routine with id=0
2022-06-12T15:01:01.937+0000 no indexes to restore for collection
2022-06-12T15:01:01.937+0000 restoring indexes for collection from metadata
2022-06-12T15:01:01.937+0000 index: &idx.IndexDocument{Options:primitive.M{"background":true, "name":"Status_EntitySubType", "v":2}, Key:primitive.D{primitive.E{Key:"Status", Value:1}, primitive.E{Key:"EntitySubType", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-06-12T15:01:01.937+0000 run create Index command for indexes: Status_EntitySubType
2022-06-12T15:01:01.937+0000 starting index build routine with id=1
2022-06-12T15:01:01.937+0000 restoring indexes for collection from metadata
2022-06-12T15:01:01.937+0000 index: &idx.IndexDocument{Options:primitive.M{"background":true, "name":"date", "v":2}, Key:primitive.D{primitive.E{Key:"SentAt", Value:-1}}, PartialFilterExpression:primitive.D(nil)}
2022-06-12T15:01:01.937+0000 index: &idx.IndexDocument{Options:primitive.M{"background":true, "name":"EntityId_NotificationId", "v":2}, Key:primitive.D{primitive.E{Key:"EntityId", Value:1}, primitive.E{Key:"NotificationId", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-06-12T15:01:01.937+0000 run create Index command for indexes: date, EntityId_NotificationId
2022-06-12T15:01:01.937+0000 restoring indexes for collection notifications.EmailLogs from metadata
2022-06-12T15:01:01.937+0000 index: &idx.IndexDocument{Options:primitive.M{"background":true, "name":"Subject", "v":2}, Key:primitive.D{primitive.E{Key:"Subject", Value:-1}}, PartialFilterExpression:primitive.D(nil)}
2022-06-12T15:01:01.937+0000 index: &idx.IndexDocument{Options:primitive.M{"background":true, "name":"SentAt", "v":2}, Key:primitive.D{primitive.E{Key:"SentAt", Value:-1}}, PartialFilterExpression:primitive.D(nil)}
2022-06-12T15:01:01.937+0000 run create Index command for indexes: Subject, SentAt
2022-06-12T15:01:01.937+0000 starting index build routine with id=2
2022-06-12T15:01:01.937+0000 no indexes to restore for collection notifications.EmailTemplates
2022-06-12T15:01:01.937+0000 restoring indexes for collection notifications.SMSLogs from metadata
2022-06-12T15:01:01.937+0000 index: &idx.IndexDocument{Options:primitive.M{"background":true, "name":"Phone_NotificationId", "v":2}, Key:primitive.D{primitive.E{Key:"Phone", Value:1}, primitive.E{Key:"NotificationId", Value:1}}, PartialFilterExpression:primitive.D(nil)}
2022-06-12T15:01:01.937+0000 run create Index command for indexes: Phone_NotificationId
2022-06-12T15:01:01.944+0000 Failed: notifications.SMSLogs: error creating indexes for notifications.SMS: createIndex error: connection() error occured during connection handshake: auth error: unable to authenticate using mechanism "SCRAM-SHA-256": (KeyNotFound) Cache Reader No keys
found for HMAC that is valid for time: { ts: Timestamp(1655046061, 8295) } with id: 0