I’m writing a Scala app that will need to handle big spikes in load. I was quite surprised to see an exception that says the “Max number of threads (maxWaitQueueSize) of 500 has been exceeded”
500! Why on earth would I need 500 threads?
The documentation for Scala driver (which apparently uses the Java driver) says that it’s non-blocking and asynchronous. I’ve always thought that these two concepts are usually implemented by reusing a limited number of threads running on a thread pool of a particular size. Upon looking into the driver code I noticed that the mongo driver is creating a thread for each operation! How can that be efficient? Thread creation is after all, to the best of my knowledge, absurdly expensive.
I anticipate that for sure there is a reason why is this is implemented this way, so I’d love if you could enlighten me on the following questions:
Why are you creating threads for each operation?
How can I work around the problem of running out of threads upon big load spikes?
Lastly, as I believe my understanding of the topic is limited, can you recommend me a resource for diving into details of performing async operations on the JVM
Could you provide the version of mongo-scala-driver you are using ? Could you also provide a code snippet on how the MongoClient is initiated ?
The MongoClient instance represents a pool of connections for a given MongoDB server deployment; you will only need one instance of class MongoClient even with multiple concurrently executing asynchronous operations.
Hi Wan,
thanks for the reply. I’m using driver version 2.6.0. Also I’m quite sure I’m only using a single instance of MongoClient:
class MongoClientFactory(val config: Configuration) extends DefaultBsonTransformers {
private val mongoClient: MongoClient = MongoClient(config.dbUrl)
private val database = mongoClient.getDatabase(config.dbName)
def getDatabase(codecProviders: Seq[CodecProvider]): MongoDatabase = {
val codecRegistry = fromRegistries(fromProviders(codecProviders: _*), DEFAULT_CODEC_REGISTRY)
database.withCodecRegistry(codecRegistry)
}
}
In the meantime I think I’ve managed to anwer my own question, but I hope you could clarify I got this right. As I stated above, I was trying to perform a lot of concurrent write operations to the DB. Creating a thread for each operation and queueing it makes sense speed wise - if we can spawn and initialize a thread before a connection pool slot becomes available we save time on the potential context switch to reuse an existing thread. Am I reasoning this correctly?
Yes, the concept is to re-use threads in the connection pool. Please be aware that there is a limit for a thread to wait for a connection to become available. ConnectionPoolSettings: maxWaitTime.
You may have already done this, but if possible it’s best to batch operations together if they can be batched. For example, using insertMany() instead of multiple insert() or utilise Bulk Write Operations. This should reduce the number of operations waiting in queue.
I get the connection pooling, and yes I am using bulk operations, but my question still stands - why is each read/write operation spawned as a separate thread? Can’t we just queue Runnables instead of Threads and supply them to existing connection pool threads from the queue? What’s the benefit here?
The Connection Pool is used as the resource of connections to send operations to MongoDB. Each operation does not necessarily spawn a new connection (thread) from the pool, but will do so if required up to the max connection pool size. If the connection pool size is at maximum and all connections are in use then pending operations are added to the wait queue. If the wait queue exceeds the configured size an error is thrown.
This pool and wait queue behaviour is the same in the sync and async version of the underlying java driver. What is different is the sync driver blocks on waiting for the result of an operation, this generally makes it less likely that app code will exhaust the pool and wait queue (although certainly not impossible).
With the async driver, it is far easier to write app code that does multiple concurrent database operations in a request. Unlike the sync version each will use a connection from the pool at the same time and that can exasperate the issue of connection exhaustion and need the wait queue.
Please note in the next major release of the Scala driver the max wait queue size limitation is being removed but until then you can configure it via the waitQueueMultiple connection string setting.
To understand fully what is going on in your scenario and to ensure nothing else is exasperating the issue I’d need to see some example code.