Does Ross Lawley (author of this plugin) pay attention to these forums?
I include below the response from Ross Lawley to my ticket for this issue on MongoDB’s Jira:
Hi Dan S,
Many thanks for the ticket. When working with multiple distributed systems, it often can be difficult to diagnose the root cause of too many cursors. I have a feel that error you are seeing is the symptom of an issue rather than the root cause. If that is the case then adding a configuration wouldn't be the right thing to do.
Could you provide more detail on how you are hitting this issue? What version of MongoDB are you running? What OS? What version of Spark and what version of the Spark connector? Ideally, a minimal reproducible example would help as I could replicate the issue.
I believe that this forum is the most appropriate place for this discussion. The database used is Amazon’s DocumentDB, which is supposed to support the same client-server protocol as the database from MongoDB. The number of open cursors allowed for any database is a fixed parameter that corresponds to the size of the EC2 instances on which the managed database is to run. see here Note that the smallest instance allows up to 30 cursors.
I have a batch job that runs a Spark application which includes reads from a MongoDB database using the Spark plugin. If there are 4 or more instances of this application running at the same time, while the database is on the smallest tier, they will produce this error. From this I reason the following:
- A single read operation appears to open >7 and <10 cursors on average to the MongoDB database, certainly not 1.
- There appears to be no mechanism to choose how many cursors the plugin opens during read operations. I am also unaware of where this number is published. This makes it impossible to reason about how many applications can simultaneously read before a failure occurs.
I may be misunderstanding, but there is an arbitrary number of cursors opened by this plugin to the database, with no mechanism to control this number.