Docs Home → MongoDB Spark Connector
Configuration Options
Various configuration options are available for the MongoDB Spark Connector. To learn more about the options you can set, see Write Configuration Options and Read Configuration Options.
Specify Configuration
Using SparkConf
You can specify configuration options with SparkConf using any of
the following approaches:
The
--confflag at runtime. To learn more, see Dynamically Loading Spark Properties in the Spark documentation.The
$SPARK_HOME/conf/spark-default.conffile.
The MongoDB Spark Connector will use the settings in SparkConf as
defaults.
Important
When setting configurations with SparkConf, you must prefix the
configuration options. Refer to Write Configuration Options and
Read Configuration Options for the specific prefixes.
Using an Options Map
In the Spark API, the DataFrameReader and DataFrameWriter methods
accept options in the form of a Map[String, String]. Options
specified this way override any corresponding settings in SparkConf.
Short-Form Syntax
Options maps support short-form syntax. You may omit the prefix when specifying an option key string.
Example
The following syntaxes are equivalent to one another:
dfw.option("spark.mongodb.write.collection", "myCollection").save()dfw.option("spark.mongodb.collection", "myCollection").save()dfw.option("collection", "myCollection").save()
Using a System Property
The connector provides a cache for MongoClients which can only be
configured with a System Property. See Cache Configuration.
Cache Configuration
The MongoConnector includes a cache for MongoClients, so workers can share the MongoClient across threads.
Important
As the cache is setup before the Spark Configuration is available, the cache can only be configured with a System Property.
System Property name | Description |
|---|---|
mongodb.keep_alive_ms | The length of time to keep a Default: |
ConfigExceptions
A configuration error throws a ConfigException. Confirm that any of
the following methods of configuration that you use are configured
properly: