Decimal values written as NumberDecimal("") - Spark Connector

Working with Data Connectors & Integrations
1

Recently, I’ve started working on writing some data to MongoDB via Spark. I have developed an application which writes to MongoDB via MongoDB Spark Connector.

I have used MongoSpark.save() to write Dataframe to MongoDB, however the decimal values are being written as NumberDecimal(""), Eg: “Score” : NumberDecimal(“0.3916”)

Instead, I would like to write as “Score” : 0.3916.

On going through articles I got to know that the Decimal is being represented as NumberDecimal, but is there a way where the data can be written without NumberDecimal() represented as like “Score” : 0.3916.

2

The quick workaround to deal with drive datatype issues is, explicit typecast the column of the dataframe in the notebook before calling save().
ex. dataframe.withColumn(field, dataframe(field).cast( DecimalType )
Please refer: (DecimalType (Spark 2.2.0 JavaDoc))))

Thanks

1 Like