I am using .net6 withdependency injection and I have single client in my app .
My connection string is like that “mongodb://localhost:27017/?maxPoolSize=1000” .
All of the threads access to mongo for lifetime.
When I use 1000 threads in my app there is no connection problem but when I use 4000 threads
I have this error; System.TimeoutException: Timed out waiting in connecting queue after 121812ms.
** at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedOrReuseAsync(CancellationToken cancellationToken)**
But when I start my app to 4 time at same time also working fine all of them
What is the problem how can I fix this problem
(I tried to increase the poolsize .Is is not working)
The maximum pool size by default is 100, so is there a reason why you want to increase it to 40x its default size? Is 100 not enough for your application? How did you determine that it is not enough?
The app has multiple task and task count is about 4000 simultaneously.
When using maxPoolSize is 100 or 1000, we get exception:
System.TimeoutException: Timed out waiting in connecting queue after 121812ms.
** at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.ConnectionCreator.CreateOpenedOrReuseAsync(CancellationToken cancellationToken)**
So we have multiple insert, update and delete operation with 4000 simultaneous operation. How we handle this? The exception said the problem occur about connection pool.
When I use 1000 threads in my app there is no connection problem but when I use 4000 threads
I have this error;
Could you clarify which one is the correct scenario? As I understand them, those two statements appear to contradict each other.
As mentioned previously, have you looked at the linked page Tuning Your Connection Pool Settings? Listed there are a series of possible scenario that you wanted to optimize for.
Additionally:
What is your deployment topology (replica set/sharded cluster) and what is the hardware spec for them
Is this an on-prem or an Atlas deployment
Are you seeing slow application-side operation times that are not reflected in the database server logs
Are you seeing high database CPU usage and the server logs show more connection attempts than expected.
Are you seeing low load on the database is low and the application performs fewer operations at any one time than expected
I tought that we must tell you our aplication how it is working
We need to read the 60K smart devices every hour.
This devices are smart electricity meters which have internet access and they can support communicaiton and data transfer
We can configure how many device we can communicating from our application at same time
Fisrt Scenario:
When we set the communicating limit to 1000 device at same time ,
our application use 1000 parallel task. Every task accesses to MongoDB at same time
According to this scenario we run our application once ,
if we set maxPoolSize:100 , the application not working so there has mongo connection error (Timed out waiting in connecting queue) .
But if we set maxPoolSize:1000, the application is working fine
Second Scenario:
if we start like first scenario again but this time we run to 4 instance our application at same time
so we provide 4000 task at same time and every thing is working fine again
but we dont want 4 instance we need to run once
Third Scenario
When we set the communicating limit to 4000 device at same time
so our application use 4000 parallel task and access to MongoDB every task at same time
According to this scenario we run our application once ,
if we set maxPoolSize:100, not working, there has mongo connection error (Timed out waiting in connecting queue)
if we set maxPoolSize:1000, not working, there has mongo connection error (Timed out waiting in connecting queue)
if we set maxPoolSize:4000, not working, there has mongo connection error (Timed out waiting in connecting queue)
As a result the problem is that; Why 1000 task and 4 instance working fine but 4000 task 1 instance don’t working
Thanks for the detailed explanation of the scenario.
However while the scenario is important, I think we also need more details about your deployment in order to understand more and possibly replicate the issue you’re seeing:
What is your deployment topology (replica set/sharded cluster), how many shards (if applicable), nodes per replica set, etc., and what is the hardware spec for them
Is this an on-prem or an Atlas deployment
Are you seeing slow application-side operation times that are not reflected in the database server logs
Are you seeing high database CPU usage and the server logs show more connection attempts than expected.
Are you seeing low load on the database is low and the application performs fewer operations at any one time than expected
In addition, if you post a short self-contained example code that can replicate this error, that would be extremely helpful in understanding it.
I also would like to confirm this part:
if we set maxPoolSize:100, not working, there has mongo connection error (Timed out waiting in connecting queue)
if we set maxPoolSize:1000, not working, there has mongo connection error (Timed out waiting in connecting queue)
if we set maxPoolSize:4000, not working, there has mongo connection error (Timed out waiting in connecting queue)
So no matter what the maxPoolSize setting (100, 1000, 4000) when communicating with 4000 devices simultaneously (I’m assuming you mean you’re trying to insert 4000 documents at the same time), it never works, but when you try using 1000 simultaneous inserts, they can work? You mentioned 1000 & 4000, have you tried something in between e.g. 2000?
In the meantime, in the absence of any information regarding your deployment, you might want to take a look at the minPoolSize setting in addition to setting maxPoolSize.