I have a 3 node mongo 4.0.27 replica set with 1 primary and 2 secondary nodes.
I am executing a Push operation on a List to add new entries. However this is resulting in an overwrite of previous entries as the $push seems to be getting converted into $set. in the oplog.
I am trying to insert 20 new entries into the list.
Can see the following in the driver DEBUG logs (partial set of logs)
2022-08-30 13:11:42 [http-nio-8080-exec-45] DEBUG org.mongodb.driver.protocol.command -Sending command ‘{“update”: “orgEntity”, “ordered”: true, “writeConcern”: {“w”: 3, “j”: true}, “txnNumber”: 59, “$db”: “productDB”, “$clusterTime”: {“clusterTime”: {"$timestamp": {“t”: 1661845302, “i”: 29}}, “signature”: {“hash”: {"$binary": {“base64”: “AAAAAAAAAAAAAAAAAAAAAAAAAAA=”, “subType”: “00”}}, “keyId”: 0}}, “lsid”: {“id”: {"$binary": {“base64”: “qAABBCCDDEEFFSSAAEE/AAD7A==”, “subType”: “04”}}}, “updates”: [{“q”: {"_id": {"$oid": “63085377d8db0b0e3cccac63”}}, “u”: {"$push": {“dBRefForLocEntity”: {"$ref": “geolocation”, “$id”: {"$oid": “630dbf36b2d3b5194196f4bd”}}}}}]}’ with request id 1059 to database productDB on connection [connectionId{localValue:8, serverValue:47}] to server mongo1:27021
2022-08-30 13:11:42 [http-nio-8080-exec-38] DEBUG org.mongodb.driver.protocol.command -Sending command ‘{“update”: “orgEntity”, “ordered”: true, “writeConcern”: {“w”: 3, “j”: true}, “txnNumber”: 27, “$db”: “productDB”, “$clusterTime”: {“clusterTime”: {"$timestamp": {“t”: 1661845302, “i”: 38}}, “signature”: {“hash”: {"$binary": {“base64”: “AAAAAAAAAAAAAAAAAAAAAAAAAAA=”, “subType”: “00”}}, “keyId”: 0}}, “lsid”: {“id”: {"$binary": {“base64”: “bpl+TemyRby34zFE5DQheQ==”, “subType”: “04”}}}, “updates”: [{“q”: {"_id": {"$oid": “63085377d8db0b0e3cccac63”}}, “u”: {"$push": {“dBRefForLocEntity”: {"$ref": “geolocation”, “$id”: {"$oid": “630dbf36b2d3b5194197025a”}}}}}]}’ with request id 1144 to database productDB on connection [connectionId{localValue:10, serverValue:48}] to server mongo1:27021
2022-08-30 13:11:43 [http-nio-8080-exec-39] DEBUG org.mongodb.driver.protocol.command -Sending command ‘{“update”: “orgEntity”, “ordered”: true, “writeConcern”: {“w”: 3, “j”: true}, “txnNumber”: 34, “$db”: “productDB”, “$clusterTime”: {“clusterTime”: {"$timestamp": {“t”: 1661845303, “i”: 2}}, “signature”: {“hash”: {"$binary": {“base64”: “AAAAAAAAAAAAAAAAAAAAAAAAAAA=”, “subType”: “00”}}, “keyId”: 0}}, “lsid”: {“id”: {"$binary": {“base64”: “bpl+TemyRby34zFE5DQheQ==”, “subType”: “04”}}}, “updates”: [{“q”: {"_id": {"$oid": “63085377d8db0b0e3cccac63”}}, “u”: {"$push": {“dBRefForLocEntity”: {"$ref": “geolocation”, “$id”: {"$oid": “630dbf37b2d3b51941970ff8”}}}}}]}’ with request id 1208 to database productDB on connection [connectionId{localValue:10, serverValue:48}] to server mongo1:27021
2022-08-30 13:11:43 [http-nio-8080-exec-41] DEBUG org.mongodb.driver.protocol.command -Sending command ‘{“update”: “orgEntity”, “ordered”: true, “writeConcern”: {“w”: 3, “j”: true}, “txnNumber”: 54, “$db”: “productDB”, “$clusterTime”: {“clusterTime”: {"$timestamp": {“t”: 1661845303, “i”: 11}}, “signature”: {“hash”: {"$binary": {“base64”: “AAAAAAAAAAAAAAAAAAAAAAAAAAA=”, “subType”: “00”}}, “keyId”: 0}}, “lsid”: {“id”: {"$binary": {“base64”: “brfjBuGQSoeu5VwXGQMo1Q==”, “subType”: “04”}}}, “updates”: [{“q”: {"_id": {"$oid": “63085377d8db0b0e3cccac63”}}, “u”: {"$push": {“dBRefForLocEntity”: {"$ref": “geolocation”, “$id”: {"$oid": “630dbf37b2d3b51941971d95”}}}}}]}’ with request id 1286 to database productDB on connection [connectionId{localValue:8, serverValue:47}] to server mongo1:27021
2022-08-30 13:11:43 [http-nio-8080-exec-37] DEBUG org.mongodb.driver.protocol.command -Sending command ‘{“update”: “orgEntity”, “ordered”: true, “writeConcern”: {“w”: 3, “j”: true}, “txnNumber”: 72, “$db”: “productDB”, “$clusterTime”: {“clusterTime”: {"$timestamp": {“t”: 1661845303, “i”: 21}}, “signature”: {“hash”: {"$binary": {“base64”: “AAAAAAAAAAAAAAAAAAAAAAAAAAA=”, “subType”: “00”}}, “keyId”: 0}}, “lsid”: {“id”: {"$binary": {“base64”: “qAABBCCDDEEFFSSAAEE/AAD7A==”, “subType”: “04”}}}, “updates”: [{“q”: {"_id": {"$oid": “63085377d8db0b0e3cccac63”}}, “u”: {"$push": {“dBRefForLocEntity”: {"$ref": “geolocation”, “$id”: {"$oid": “630dbf37b2d3b51941972b33”}}}}}]}’ with request id 1367 to database productDB on connection [connectionId{localValue:10, serverValue:48}] to server mongo1:27021
From above we can see that the driver is sending $push command to the DB.
From the updatelog generated from oplog by running below commands can see that there are multiple overwrites to the same index.
“o”: {
“$v”: 1,
“$set”: {
“dBRefForLocEntity.3468”: {
“$ref”: “geolocation”,
“$id”: {
“$oid”: “630db2b16c6f5e01638234a6”
}
}
}
}
}
cat updatelog.json | grep “dBRefForLocEntity.3468”
“dBRefForLocEntity.3468”: {
“dBRefForLocEntity.3468”: {
“dBRefForLocEntity.3468”: {
“dBRefForLocEntity.3468”: {
“dBRefForLocEntity.3468”: {
“dBRefForLocEntity.3468”: {
“dBRefForLocEntity.3468”: {
“dBRefForLocEntity.3468”: {
I am using below commands to generate oplog.
mongodump -d local -c oplog.rs -o oplogD --port 27021
cd oplogD/local/
bsondump oplog.rs.bson | jq . > updatelog.json
This issue is not seen initially. After the list size grows to some 1500+ entries the issue happens consistently.
I tried all options from the code perspective (tried out all APIs provided by mongocollection/repository/$push/$addtoSet etc) but always hitting this issue.
Please provide your inputs to debug/rootcause this issue which is causing the overwrites to the list.