Spark connector skipping update events made on the same document (while reading stream from MongoDB)

While experimenting with Spark connector I noticed that when changes to a single document in MongoDB are made very fast, only the latest version of a document will be sent to Spark connector. In such cases, very often number of events sent to Spark is correct, but all events contain the latest version of a given document. It that a normal behavior? How can I avoid it, and have the whole change history sent to Spark?

Technical details

I’m using pyspark 3.3.1, connector mongo-spark-connector_2.12:10.1.1, Mongodb 4.4.13
I have spark.mongodb.change.stream.publish.full.document.only property set to True.

Example

Sample code can be found here.

For instance, say I’m modifying a collection in Mongo that consists only of docs like (_id, version).
Running 2 separate update operations in a row (updating version from 1 to 2, and afterwards from 2 to 3), without any delay, Spark in a single batch receives:

-------------------------------------------
Batch: 10
-------------------------------------------
+------------------------+-------+
|_id                     |version|
+------------------------+-------+
|63eaa487426f9a7396ba6199|3      |
|63eaa487426f9a7396ba6199|3      |
+------------------------+-------+

While if i make a small delay in between, Spark will be sent separate events instead:

-------------------------------------------
Batch: 20
-------------------------------------------
+------------------------+-------+
|_id                     |version|
+------------------------+-------+
|63eaa487426f9a7396ba6199|4      |
+------------------------+-------+

-------------------------------------------
Batch: 22
-------------------------------------------
+------------------------+-------+
|_id                     |version|
+------------------------+-------+
|63eaa487426f9a7396ba6199|5      |
+------------------------+-------+