Which version of MongoDB are you using? I found an internal ticket that says this might have been resolved in the latest versions of MongoDB and mongorestore.
Can you please confirm you are still experiencing the same issue with
mongorestore 100.1.1
MongoDB 4.4.1
If that’s not the case, please just make sure that your cluster topology stays the same during your restore operations.
Also, please note that mongodump/mongorestore cannot be part of a backup strategy for 4.2+ sharded clusters that have sharded transactions in progress. More details here.