One way I can think of is to throttle the insertion by splitting the 1 insertMany of 500_000 documents into many insertMany of fewer documents. For example, 5 insertMany of 100_000 documents with a small delay between each. You may try to adjust the delay to match the blocking time of the 1 insertMany. For example, if 1 insertMany of 500_000 blocks the server for 5 seconds, wait 1 secs between each of the 5 insertMany. This way other work can be perform in between.
But like any performance issue, the operation you perform is one factor but it also depends on the system characteristics. Because you may throttle as you wish but if you do not have the resources to handle the work load, there is nothing you can do.
This is a great workaround, and i will implement it to see how well it works, thanks for the idea
About system characteristics, we don’t see particular ressource usage when mongodb ingest the 500k documents. We tested on a server (docker + replicaset) and local instance of mongodb (no docker, no replicaset). Would you have an idea where the bottleneck is comming from ?