Docs Menu

Docs HomeMongoDB Spark Connector

Configuration Options

On this page

  • Specify Configuration
  • Cache Configuration
  • ConfigExceptions

Various configuration options are available for the MongoDB Spark Connector. To learn more about the options you can set, see Write Configuration Options and Read Configuration Options.

You can specify configuration options with SparkConf using any of the following approaches:

  • The --conf flag at runtime. To learn more, see Dynamically Loading Spark Properties in the Spark documentation.

  • The $SPARK_HOME/conf/spark-default.conf file.

The MongoDB Spark Connector will use the settings in SparkConf as defaults.

Important

When setting configurations with SparkConf, you must prefix the configuration options. Refer to Write Configuration Options and Read Configuration Options for the specific prefixes.

In the Spark API, the DataFrameReader and DataFrameWriter methods accept options in the form of a Map[String, String]. Options specified this way override any corresponding settings in SparkConf.

Options maps support short-form syntax. You may omit the prefix when specifying an option key string.

Example

The following syntaxes are equivalent to one another:

  • dfw.option("spark.mongodb.write.collection", "myCollection").save()

  • dfw.option("spark.mongodb.collection", "myCollection").save()

  • dfw.option("collection", "myCollection").save()

The connector provides a cache for MongoClients which can only be configured with a System Property. See Cache Configuration.

The MongoConnector includes a cache for MongoClients, so workers can share the MongoClient across threads.

Important

As the cache is setup before the Spark Configuration is available, the cache can only be configured with a System Property.

System Property name
Description
mongodb.keep_alive_ms

The length of time to keep a MongoClient available for sharing.

Default: 5000

A configuration error throws a ConfigException. Confirm that any of the following methods of configuration that you use are configured properly:

←  MongoDB Connector for SparkWrite Configuration Options →
Share Feedback
© 2023 MongoDB, Inc.

About

  • Careers
  • Investor Relations
  • Legal Notices
  • Privacy Notices
  • Security Information
  • Trust Center
© 2023 MongoDB, Inc.