Docs Home → MongoDB Spark Connector
Write Configuration Options
Write Configuration
The following options for writing to MongoDB are available:
Note
If you use SparkConf
to set the connector's write configurations,
prefix spark.mongodb.write.
to each property.
Property name | Description | |
---|---|---|
mongoClientFactory | MongoClientFactory configuration key. You can specify a custom implementation which must implement the
com.mongodb.spark.sql.connector.connection.MongoClientFactory
interface.Default: com.mongodb.spark.sql.connector.connection.DefaultMongoClientFactory | |
connection.uri | Required. The connection string configuration key. Default: mongodb://localhost:27017/ | |
database | Required. The database name configuration. | |
collection | Required. The collection name configuration. | |
maxBatchSize | Specifies the maximum number of operations to batch in bulk
operations. Default: 512 | |
ordered | Specifies whether to perform ordered bulk operations. Default: true | |
operationType | Specifies the type of write operation to perform. You can set
this to one of the following values:
Default: replace | |
idFieldList | Field or list of fields by which to split the collection data. To
specify more than one field, separate them using a comma as shown
in the following example:
Default: _id | |
writeConcern.w | Specifies w , a write concern option to acknowledge the level to
which the change propagated in the MongoDB replica set. You can
specify one of the following values:
For more information on w values, see the MongoDB server guide on
the WriteConcern w option.Default: ACKNOWLEDGED | |
writeConcern.journal | Specifies j , a write concern option to enable request for
acknowledgment that the data is confirmed on on-disk journal for
the criteria specified in the w option. You can specify
either true or false .For more information on j values, see the MongoDB server
guide on the
WriteConcern j option. | |
writeConcern.wTimeoutMS | Specifies wTimeoutMS , a write concern option to return an error
when a write operation exceeds the number of milliseconds. If you
use this optional setting, you must specify a non-negative integer.For more information on wTimeoutMS values, see the MongoDB server
guide on the
WriteConcern wtimeout option. | |
upsertDocument | When true , replace and update operations will insert the data
if no match exists.For time series collections, you must set upsertDocument to
false .Default: true |
connection.uri
Configuration Setting
You can set all Write Configuration via the write connection.uri
.
Note
If you use SparkConf
to set the connector's write configurations,
prefix spark.mongodb.write.
to the setting.
spark.mongodb.write.connection.uri=mongodb://127.0.0.1/test.myCollection
The configuration corresponds to the following separate configuration settings:
spark.mongodb.write.connection.uri=mongodb://127.0.0.1/ spark.mongodb.write.database=test spark.mongodb.write.collection=myCollection
If you specify a setting both in the connection.uri
and in a separate
configuration, the connection.uri
setting overrides the separate
setting. For example, given the following configuration, the
database for the connection is foobar
:
spark.mongodb.write.connection.uri=mongodb://127.0.0.1/foobar spark.mongodb.write.database=bar