Delay after connection pool got cleared

JRB · December 18, 2023, 9:55am

We have a service written in NodeJS using MongoDB Atlas. We had a situation where the service was running and then the DB user password was changed. What we have seen in our logs for the connection pool is the following:

connection checkout started
connection failed due to auth error
connection checkout failed
connection pool got cleared
after 10 seconds (!), connection pool was ready again
this was repeated 5 times until the DB access failed

We are pretty sure that the delay and the retries are not from our code, therefore my question:

is the delay between “connection pool cleared” and “connection pool ready” expected?
if yes, is it configurable?
where do the retries come from?

Many thanks in advance!

alexbevi · December 18, 2023, 2:50pm

@JRB if you changed the database user’s credentials that were initially provided to the application via the connection string, the application would have no way of discovering this without an updated connection string. The MongoClient would then need to be recreated with these updated details for the associated connection pool(s) to begin creating/checking in connections with this information.

The delays would all be attributed to events described in the Connection Monitoring and Pooling specification - not something you’re doing explicitly in code.

chris · December 18, 2023, 3:00pm

You’ll never guess what I was just looking at and thinking exactly who would know what section it was in

JRB · December 18, 2023, 9:23pm

Thanks for the link, interesting read

I think I now understand what happens:

the retries come from our code, this is not related to the mongo driver (did not realize this before)
we have not set waitQueueTimeoutMS (otherwise, we would fail faster)
an auth error occurs
SDAM (1) clears the connection pool and (2) marks the server as unknown (https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#why-mark-a-server-unknown-after-an-auth-error)
the server will only be rediscovered with the next heartbeat
default heartbeatFrequencyMS is 10 seconds (https://github.com/mongodb/specifications/blob/master/source/server-discovery-and-monitoring/server-discovery-and-monitoring.rst#heartbeatfrequencyms)
it takes up to 10 seconds until the server is rediscovered and the connection pool is marked as ready again

Would be great if I could get a confirmation for this (or tell me where I am wrong).

I think the main reason why we were running in such huge delays was to not having set waitQueueTimeoutMS. We should fail fast in case there are is some DB issue, not wait forever!