MongoDB atlas version is version 6.x and 8.x.
Spark version used is 3.2,3.3.x in Azure job databricks cluster.
Previously we used MongoDB spark connector version 2.4.1 and our spark job is being completed in 10 mins. Similarly with same code and compute, with MongoDB spark connector v10.0.5,v10.4.1 we observed our spark is taking 2.5 times longer i.e it is taking 28 minutes.
Have gone through the previously raised bug “https://jira.mongodb.org/browse/SPARK-419?filter=-4&jql=text%20~%20"spark"%20order%20by%20created%20DESC”, it was mentioned that it got fixed in 10.2.2, but couldn’t find it is fixed and validated and currently I am seeing the similar issue with latest MongoDB spark connector versions v10.x. In addition to above based on spark connector official documentation for Spark 3.1 to 3.5 versions , MongoDB connector version 2.4.x is not the standard way to use. Need help in understanding the cause for performance issues and if we can use 2.x version for latest spark versions, or if aything breaks in future ? Thank you for the help in advance.