Pyspark spark-submit unable to read from Mongo Atlas serverless(can read from free version)

I’ve been using Apache Spark(pyspark) to read from MongoDB Atlas, I’ve a shared(free) cluster - which has a limit of 512 MB storage I’m trying to migrate to serverless, but somehow unable to connect to the serverless instance - error

pyspark.sql.utils.IllegalArgumentException: requirement failed: Invalid uri: 'mongodb+srv://vani:<password>'

Pls note : I’m able to connect to the instance using pymongo, but not using pyspark.

Here is the pyspark code (Not Working):

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("MongoDB operations").getOrCreate()
print(" spark ", spark)

# cluster0 - is the free version, and i'm able to connect to this
# mongoConnUri = "mongodb+srv://"

mongoConnUri = "mongodb+srv://"

mongoDB = "versa"
collection = "name_map_unique_ip"

df =\
     .format("mongo") \
     .option("uri", mongoConnUri) \
     .option("database", mongoDB) \
     .option("collection", collection) \

Error :

22/07/26 12:25:36 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir.
22/07/26 12:25:36 INFO SharedState: Warehouse path is 'file:/Users/karanalang/PycharmProjects/Versa-composer-mongo/composer_dags/spark-warehouse'.
 spark  <pyspark.sql.session.SparkSession object at 0x7fa1d8b9d5e0>
Traceback (most recent call last):
  File "/Users/karanalang/PycharmProjects/Kafka/python_mongo/", line 30, in <module>
    df =\
  File "/Users/karanalang/Documents/Technology/spark-3.2.0-bin-hadoop3.2/python/lib/", line 164, in load
  File "/Users/karanalang/Documents/Technology/spark-3.2.0-bin-hadoop3.2/python/lib/", line 1309, in __call__
  File "/Users/karanalang/Documents/Technology/spark-3.2.0-bin-hadoop3.2/python/lib/", line 117, in deco
pyspark.sql.utils.IllegalArgumentException: requirement failed: Invalid uri: 'mongodb+srv://'
22/07/26 12:25:36 INFO SparkContext: Invoking stop() from shutdown hook
22/07/26 12:25:36 INFO SparkUI: Stopped Spark web UI at

pymongo code (am able to connect using the same uri):

from pymongo import MongoClient

client = MongoClient("mongodb+srv://")

all_dbs = client.list_database_names()
print(f"all_dbs : {all_dbs}")

Spark-submit command :

spark-submit --packages org.mongodb.spark:mongo-spark-connector:10.0.2 ~/PycharmProjects/Kafka/python_mongo/

any ideas how to debug/fix this ?


This is working code with an Atlas Serverless instance. I am using version 10.0 of the Spark Connector. Note that the conf parameter names are different than 3.x of the connector as well.

from pyspark.sql import SparkSession

spark = SparkSession.\
config("spark.executor.memory", "1g").\
config("spark.jars.packages", "org.mongodb.spark:mongo-spark-connector:10.0.3").\

df ="mongodb").option('database', 'MyDB').option('collection', 'MyCollection').load()

You can use both versions of the connector side by side, the 3.x version is “mongo” and the 10.x uses “mongodb”