Hi, I got my URI from Mongo Atlas and I just have to put my password and name of database and collection, but even with that, it keeps giving me this Invalid uri
error. I tried changing my password but still I cannot write into my MongoDB collection.
I have the serverless modality if that’s relevant.
This is the uri I’m using (the one given to me by Atlas, password not included):
mongodb+srv://jmcmt87:<password>@twittermongoinstance.db1xm.mongodb.net/twitter_data.aggregated_data?retryWrites=true&w=majority
And in case it’s relevant, this is my configuration:
packages = ','.join([
'org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1',
'com.amazonaws:aws-java-sdk:1.11.563',
'org.apache.hadoop:hadoop-aws:3.2.2',
'org.apache.hadoop:hadoop-client-api:3.2.2',
'org.apache.hadoop:hadoop-client-runtime:3.2.2',
'org.apache.hadoop:hadoop-yarn-server-web-proxy:3.2.2',
'com.johnsnowlabs.nlp:spark-nlp-spark32_2.12:3.4.2',
'org.mongodb.spark:mongo-spark-connector_2.12:3.0.1'
])
spark = SparkSession.builder.appName('twitter_app_nlp')\
.master("local[*]")\
.config('spark.jars.packages', packages) \
.config('spark.streaming.stopGracefullyOnShutdown', 'true')\
.config('spark.hadoop.fs.s3a.aws.credentials.provider',
'org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider') \
.config('spark.hadoop.fs.s3a.access.key', ACCESS_KEY) \
.config('spark.hadoop.fs.s3a.secret.key', SECRET_ACCESS_KEY) \
.config("spark.hadoop.fs.s3a.impl",
"org.apache.hadoop.fs.s3a.S3AFileSystem") \
.config('spark.sql.shuffle.partitions', 3) \
.config("spark.driver.memory","8G")\
.config("spark.driver.maxResultSize", "0") \
.config("spark.kryoserializer.buffer.max", "2000M")\
.config("spark.mongodb.input.uri", mongoDB) \
.config("spark.mongodb.output.uri", mongoDB) \
.getOrCreate()
Where mongDB variable is the string of the uri aforementioned.
What could be the problem?