Hi,
Maybe you didn’t set the timeout to a big enough value? The latency shouldn’t make a difference I think. If I’m not mistaking, if the primary can’t be found, but you are trying to write, with retryable writes ON, you shouldn’t get an error until you reached the timeout for this query because the “retry” operation will only happen once - but will wait for a primary to be present before retrying. If you get another error at that point, you get the exception, there is no second retry operation. But the latency shouldn’t make a difference, as long as the total time is lower than your timeout, you should be good, in theory.
If I remember correctly, I was barely able to do 1500 updates during the 6 min failover test I did. I think your code is using promises so it’s probably async, no? Maybe that makes a difference as you may be queuing “too many” queries during the failover which could then timeout ? I’m not sure. 
.
In any case, you should always protect your code with a try/catch mechanism. Maybe someday the network will fail completely and you won’t have a primary for 5 minutes. If a command is retried successfully, you won’t get an exception, it’s transparent. But if it fails twice or timeout, you need to be able to handle this case too.
Cheers,
Maxime.