Aggregation pipeline not working in spark mongo connector v10

I am trying to filter data using aggregation pipeline. In spark mongo connector v3.0, it is working and in V10, the filter is not applied and all documents from collection is returned

v3.0 code: (filter is working)

spark.read.format("mongo").option("uri",_uri).option("database",_database) \
                .option("collection", _collection).option("pipeline","{$match:{flag:0}}").load()

v10 code : (not working)

spark.read.format("mongodb").option("spark.mongodb.connection.uri",_uri) \
                    .option("spark.mongodb.database",_database) \
                    .option("spark.mongodb.collection", _collection)\
                    .option("pipeline","{$match:{flag:0}}")\
                    .load()

Can you anyone please help me on this?

1 Like

It worked. I have used

spark.read.format(“mongodb”).option(“spark.mongodb.connection.uri”,_uri)
.option(“spark.mongodb.database”,_database)
.option(“spark.mongodb.collection”, _collection)
.option(“aggregation.pipeline”,"{$match:{flag:0}}")
.load()

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.