Getting timeout error when deleting data from colletion

Working with Data
#1

Hi,

I am using MongoDB.Driver (.net nuget package) and having around 1 lakh documents in DB.

I am deleting multiple BsonDocuement in the batches of 1500 ObjectIds using this filter

Builders<BsonDocument>.Filter.In("_id", id);

I am getting this below error

Found exception One or more errors occurred. (A write operation resulted in an error. WriteError: { Category: "ExecutionTimeout", Code: 50, Message: "Request timed out." }

At first, the time when I run code, it deletes 1500 records successfully for a few batches around 20-25 but for later batches, it gives a timeout. we have 1600 RUs in the database.

protected FilterDefinition<BsonDocument> ItemWithListOfId(List<ObjectId> id){
            return Builders<BsonDocument>.Filter.In("_id", id)}

Parallel.ForEach(_batch, list =>
{
  try
  {
      DeleteResult result = collection.DeleteManyAsync(ItemWithListOfId(ids));
      if (result.IsAcknowledged)
      {
          totalDeleteRecords = totalDeleteRecords + (Int32)result.DeletedCount;
          
      }
  }
  catch (Exception ex)
  {
      throw ex;
  }});

Can anyone help me to fix this issue, I can further share more info if needed.

#3

Did you notice anything on your dashboard when you get that error?
What write concern are you using? in case of majority, did the replication succeed?

#4

hi @Kobe_W

Nothing suspicious at dashboard, and i am not using any write concern.

#5

Hello :wave: @Abhishek_bahadur_shrivastav,

Welcome to the MongoDB Community forums :sparkles:

What I understand is that this query will be translated into

db.test.deleteMany({_id:{$in:[1,2,3...]}})

which means that there will be one deleteMany() command, but the $in array will have 1500 entries. Let me know if my understanding is correct here.

Could you please clarify the meaning of RUs in this context, and also let me know where you have deployed the MongoDB server?

Here, the Execution timeouts mean the entire task ran for longer than the configured amount of time.

Could you kindly share the total number of documents in your collection and the percentage that you intend to delete?

If the percentage of documents to be deleted is high, you can use the aggregation pipeline with $match and $out to write the documents you want to keep to a different collection (assuming this is not a sharded cluster). Afterward, drop the original collection and rename the new collection to retain the desired documents.

However, there could be a couple of workarounds that can be considered:

  • One approach is to raise the batch size from 1500 to 10000, which may enable the operation to complete within the default time limit.

  • Alternatively, you may want to adjust the settings for MongoServerSettings.OperationTimeout to fine-tune the default OperationTime.

I hope it helps!

Best,
Kushagra

#6

hi :wave: @Kushagra_Kesav ,

Thanks for reply.

Your understanding is correct. I am sending 2 parallel request (each request has 1500 ObjectId’s) to delete BsonDocument asynchronously.

db.test.deleteMany({_id:{$in:[1,2,3...]}})

I have Azure CosmosDB db via mongo API and having more than 10 lakh records in each collections and RUs means [$Request unit] (Request Units as a throughput and performance currency - Azure Cosmos DB | Microsoft Learn).

Could you please clarify the meaning of RUs in this context, and also let me know where you have deployed the MongoDB server?

How can i verify that percentage of deletion is high than expected in my database. I am not much familiar about shared cluster. our use case is to clean up our database if few Bsondocument are not in use. we can’t drop existing collection and create new one.

If the percentage of documents to be deleted is high, you can use the aggregation pipeline with $match and $out to write the documents you want to keep to a different collection (assuming this is not a sharded cluster ). Afterward, drop the original collection and rename the new collection to retain the desired documents.

I tried to keep then 500 Object-Id’s in each request but still its giving execution timeout.

  • One approach is to raise the batch size from 1500 to 10000, which may enable the operation to complete within the default time limit.

how can we use this option through MongoDB.Driver nuget package, since we are setting up connection via MongoClientSettings from our code.

It would be more fruitful for me if we meet on call, if possible. Please schedule a call at my booking page
Book with Abhishek. i can explain my problem better and find the best solution together.

thanks @Kushagra_Kesav