How To SELECT ... FOR UPDATE inside MongoDB Transactions

The most common question that I receive as a MongoDB Consultant since we introduced Multi-document ACID transactions in MongoDB 4.0 is: "How can I be sure that the documents that I am reading inside a transaction are not modified by another operation before the commit?"

Before answering this question, I would first like to explain how concurrency works in MongoDB, inside and outside of transactions. Knowing this helps understand the answer.

Locking

MongoDB uses multi-granularity locking that allow shared access to a resource, such as a database or collection. There is a detailed description of the types of locks in our documentation.

All reads and writes performed as part transaction will need to take an intent exclusive (IX) lock on the collection they are accessing. Sometimes that's not possible because another operation has an exclusive or shared lock (X or S) on that collection or its database, in which case to avoid possible deadlock, the transaction only waits for 5ms and aborts otherwise. The 5ms is the default and can be changed via maxTransactionLockRequestTimeoutMillis. However, operations that take an exclusive or shared lock on a collection are not very common, so let's look at what's more likely to create a conflict for your transaction.

Write conflicts and Transactions

In WiredTiger, an update writes to a document optimistically and aborts if a conflicting update is detected, as it will in certain other cases, such as excessive memory pressure. That's called writeConflict because that's the name of the type of exception it throws. Outside of transactions, such writes will be retried again automatically inside MongoDB until they succeed.

The scenario is a bit different in case of transactions. Let's consider the following examples:

A transaction (t1) can be subject to a writeConflict if another write operation modifies the same document (D1) after the transaction starts and before the transaction itself tries to modify it. This can happen whether or not the other write operation is in a transaction. This is shown in the left image as an independent statement, and in the right image as a second transaction (t2).

Here is an example of a writeConflict error as seen from the mongodb shell:

> coll1.update( { _id: 1 },{ $set: { a: 5 } });
WriteCommandError({
   "errorLabels" : [
       "TransientTransactionError"
   ],
   "operationTime" : Timestamp(1566806753, 1),
   "ok" : 0,
   "errmsg" : "WriteConflict",
   "code" : 112,
   "codeName" : "WriteConflict",
   "$clusterTime" : {
       "clusterTime" : Timestamp(1566806753, 1),
       "signature" : {
           "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
           "keyId" : NumberLong(0)
       }
   }
})

In case a writeConflict is raised inside a transaction the storage engine will not automatically retry but it will return the error to the driver.

When a writeConflict is returned, the driver can safely retry the entire transaction using our new transaction callback API that allows to automatically retry in case of TransientTransactionError (such as writeConflicts). For more details on our new callback API please refer to your driver documentation.

Write operations outside the transaction will automatically retry in the storage engine on writeConflict on a document that's been modified inside a transaction until it succeeds.

The Read Answer

Now that we know what happens every time we perform write operations inside a transaction, we can focus on what happens when we perform read operations.

When a transaction is started with read concern snapshot, we are guaranteed to read from a consistent point in time at the cluster level. The data that we are reading inside the transaction is not affected by any write operation that happens outside the transaction.

For example, when transaction t1 starts it gets a snapshot of the data, including document D1 which it plans to read. Meanwhile transaction t2 has its own snapshot and it deletes document D1. Transaction t1 can still read document D1 though because it is referring to its snapshot and is therefore isolated from other transactions write operations.

What this means, practically, is that we can't be sure that after we read documents inside a transaction, those documents won't be changed by another operation for the duration of our transaction.

In the relational world this is resolved by using SELECT...FOR UPDATE. This statement allows locking rows we are reading as though they were updated, preventing other operations from modifying or deleting them until our transaction ends.

The same behaviour can be reproduced in MongoDB by modifying a document so other transactions that try to write to it will receive a writeConflict exception.

But what should we update? We can't just rewrite an existing field's value with its current value because MongoDB is efficient and will not modify a document if a write does not actually change the existing value.

Consider the following example, in the collection we have the document:

{ _id: 1, status: true }

And we perform the following update inside the transaction:

db.foo.update({ _id: 1 }, { $set: { status: true } })
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 0 })

Since we didn't actually modify the document (status was already true), other operations can modify the document without running into any writeConflict.

Emulating a "Write Lock" while Reading

To ensure we prevent other operations from modifying our document, we have to make sure that we are writing a new value to our document. Usually we don't have a field that we can just change to make a write happen, but MongoDB's flexible schema makes it easy for us to set an attribute we're going to use exclusively to acquire our "write lock".

The next question is what value should we set our new attribute to? It needs to be a value that we know will be different from whatever value may already be in the field. We already have a data type in MongoDB with those properties: ObjectId. Since ObjectId is generated based on a combination of the Unix Epoch, random values and a counter, it's extremely unlikely for the same value to be generated more than once. That means when we update it, the update will set the field to a different value than what's already there. Once the update happens, other operations will get a writeConflict.

What does that look like in code? Like this:

var doc = db.foo.findOneAndUpdate({ _id: 1 },
     { $set: { myLock: { appName: "myApp", pseudoRandom: ObjectId() } } })

The beauty of this method is that no extra step is needed to "unlock" the document - it will happen automatically when you commit or abort the transaction. The value of the locking field - in our example, myLock - can be anything. It does not matter what the value is, as long as it changes the existing value. Using findOneAndUpdate also keeps the number of round trips to the database down to one as it both updates the locking field in the document and returns the document to the application.

The value of the locking field does not need to be read by any transaction. However, you can use the locking field to store information on which application is locking the thread. In our example, myLock is an object with the name of the application taking the lock and the locking field.

In Closing

The ability to lock documents through MongoDB's transaction support is a powerful aid to ensuring consistency. Remember, even though MongoDB supports transactions, all our existing best practices are still valid and you should use transactions only when needed. You should always look to leverage the rich and flexible MongoDB document format which in many cases eliminates the need to use transactions at all.