$inc fails on client but MongoDB processes it anyway

Working with Data Drivers
,
1

I’m working on replacing mgo with the official MongoDB driver for Go, and one of my tests is randomly breaking in an unexpected way. Here what the test does:

  1. Create a proxy (with toxiproxy) to a local MongoDB instance.
  2. Disable the proxy, so the database seems to be down.
  3. Run a function that does an update that increments a field. If it fails, it keeps retrying until the command eventually succeeds.
  4. Enable the proxy.
  5. Wait for the function to complete and assert that the field has been incremented correctly.

This test randomly breaks because sometimes that field gets incremented twice. I noticed that it happens when an update is retried just as the proxy gets enabled: the client code receives an incomplete read of message header: context deadline exceeded error, which makes it retry the command, but the previous one indeed succeeded because the field ends up being incremented twice.

Is there anything that I can do on my side to prevent this from happening? I tried to find a specific error to catch, but I couldn’t find any. Or is this something the driver itself should handle? Any help is appreciated.

2 Likes
2

Hey @mbenford, thanks for the question! To get the behavior you’re looking for, I believe you need to use a transaction.

Here’s an example of how to start a session and then complete an UpdateOne increment operation in a transaction. If WithTransaction returns an error, the transaction was not committed and can be retried.

sess, err := client.StartSession()
if err != nil {
	panic(err)
}
defer sess.EndSession(context.TODO())

_, err = sess.WithTransaction(
	context.TODO(),
	func(sessCtx mongo.SessionContext) (interface{}, error) {
		coll := client.Database("").Collection("")
		filter := bson.D{{"a", "b"}}
		update := bson.D{{"$inc", bson.D{{"x", 1}}}}
		return coll.UpdateOne(sessCtx, filter, update)
	})
if err != nil {
	panic(err)
}

See a full example:

P.S. toxiproxy sounds like a really useful tool for introducing network faults!

1 Like