Implementing locking in transaction for read AND write

How really locking and transaction works with concurrent request in mongodb? So for that I decide to implement one algorithm. The algorithm is get count from A collection, if count less than 5 then insert document.
I am using nodejs with mongodb. I went through mongo concurrency article (https://docs.mongodb.com/manual/faq/concurrency/) and transactions in mongodb, but cannot find any sample coding solution with locking.
I have achieved solution with Optimistic concurrency control (when there is low contention for the resource - This can be easily implemented using versionNumber or timeStamp field).
I will be happy if u suggest any solution.

I have posted on stackoverflow as well but no reply yet.

Hi @Naisarg_Parmar and welcome in the MongoDB Community :muscle: !

MongoDB supports multi-document ACID transactions since 4.0 which is the solution to the use case explained here. Check out Lauren’s blog post here for NodeJS: How to Use MongoDB Transactions in Node.js | MongoDB Blog. Or mine with Java that covers a similar use case: “Don’t sell more than you have in stock”: Java and MongoDB 4.0 Support for Multi-Document ACID Transactions | MongoDB Blog

It’s easier to chat here than on SO. Please feel free to continue the conversation if my answer isn’t clear enough.

Cheers,
Maxime.

2 Likes

Thank you @MaBeuLux88_xxx. Let me explore you answer and implement it. Later, I will come back.

1 Like

Hi @MaBeuLux88_xxx. I have implemented transaction but still cannot handle concurrent request. I went through your suggested article and youtube mongo ACID Transaction videos as well.
I will be happy if you please help me by looking at the code https://github.com/naisargparmar/concurrencyMongo.
Code is very very small. Sorry, I am asking you to look at the code. But I really appreciate if u take out some time for this.
Thank you Maxime

Hi @Naisarg_Parmar,

Code is great! I’ll try to have a look. In the meantime I noticed that you are missing the package.json file that usually contains the list of packages you are using.

Could you also provide me with a “test” that fails and shows what isn’t working in this code? I guess this could be a set of CURL commands in a bash script for example as you are using a REST API here.

Cheers,
Maxime.

Thank you for looking at github repository.
I am running the project without package.json. Anyhow, I will create package.json and upload to git. Meanwhile, I have mentioned how to setup in readme file. U just need to clone and run “npm i mongodb”, “npm i express” and “node app.js” command in terminal.
For testing, I am using jmeter to send multiple request at same time. The endpoint which I have mention in readme file, require json body for example {email: “em@xyz.com”}. Api code in transaction simply get the count of bookings collection. if count is <5 then insertOne will execute.
Now the issue which I am facing is, jmeter is set to send 10 request at same time than 10 insert query is executed. According to my code done in transaction, I expect the behaviour, after 5 insert, other 5 should not allow to insert.
If possible then I can live chat and explain you at some other place.

Can you share the jmeter script and explain how I can run it so I can reproduce the problem?

Note: You need to install jmeter.

For Jmeter testing look my git repository. I have uploaded one illustrating video, it shows, how I am sending multiple request with json body. I have added all files related to jmeter testing in the repo. Only follow my video, no other setup is required.

package.json is also added to the project.
So now to run project, just download/clone folder from git. Open terminal. run command “npm install” and “npm start”.
Note: Please refer readme file for further instructions .

1 Like

Hey @Naisarg_Parmar,

OK I spent a couple of hours on this and I think I understand the problem.
From what I see, you are trying to limit the number of documents in this collection to 5 and you tried to secure this in a transaction to ensure this - but the logic is flawed.

Let me explain what I have on my side and how I came to this conclusion. I wasn’t able to get Jmeter to work, but I created this little bash script to send 10 POST queries in parallel.

#!/usr/bin/env bash
seq 1 10 | xargs -n1 -P10 curl -X POST 'http://localhost:3001/insert' -H 'Content-Type: application/json' -d '{"doc": "Max"}'

When I execute this script, here is what I get in the server logs:

Server is running on PORT 3001
Database connected
body:  { doc: 'Max' }
body:  { doc: 'Max' }
body:  { doc: 'Max' }
body:  { doc: 'Max' }
body:  { doc: 'Max' }
body:  { doc: 'Max' }
body:  { doc: 'Max' }
body:  { doc: 'Max' }
body:  { doc: 'Max' }
body:  { doc: 'Max' }
1 booking_count:  0
1 booking_count:  0
1 booking_count:  0
1 booking_count:  0
1 booking_count:  0
1 booking_count:  0
1 booking_count:  0
1 booking_count:  0
1 booking_count:  0
1 booking_count:  0
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfa9")
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfaa")
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfab")
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfac")
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfad")
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfae")
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfaf")
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfb0")
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfb1")
2 insertedRecord:  new ObjectId("616b6048d6ae6b6f3d7acfb2")
Transaction Success
Transaction Success
Transaction Success
Transaction Success
Transaction Success
Transaction Success
Transaction Success
Transaction Success
Transaction Success
Transaction Success

As you can see, the 10 requests where all received exactly at the same time and they all entered in their own transactions at the same time.

Within EACH transaction, you read the number of document in this collection. They all read zero as nothing was inserted yet. From this point on, it means that your 10 transactions are valid and you can insert your document as you can pass in the true branch of the IF statement.

Result: ALL the transactions succeed and you end up with 10 documents in your database.

Now that this is done and these transactions are committed, if I try the same thing again, the 10 transactions are aborted as the document count is now equal to 10.

The code in its current state “works”. But it’s not a valid implementation to limit the number of documents in a collection.
The reason is that you cannot read what is happening in another transaction running in parallel. That’s the principe of read isolation and there is no way to go around that as far as I know.

Basically, each transaction is self standing and should be self sufficient. Here, you are trying to access data outside the transaction and that just cannot work.

So in the end, I don’t know how to solve your problem as I’m not sure what problem you are trying to solve.

If you are trying to limit the number of documents in a collection to 5 though, I would use a capped collection:

test [direct: primary] test> db.bookings.drop()
true
test [direct: primary] test> db.createCollection("bookings", {capped:true, size:999999999, max:5})
{ ok: 1 }
test [direct: primary] test> for (let i=0; i<7; i++){ db.bookings.insertOne({nb: i})};

test [direct: primary] test> db.bookings.find()
[
  { _id: ObjectId("616b6717fa19ded77023e3bc"), nb: 2 },
  { _id: ObjectId("616b6717fa19ded77023e3bd"), nb: 3 },
  { _id: ObjectId("616b6717fa19ded77023e3be"), nb: 4 },
  { _id: ObjectId("616b6717fa19ded77023e3bf"), nb: 5 },
  { _id: ObjectId("616b6717fa19ded77023e3c0"), nb: 6 }
]

But note that you cannot write in capped collections in transactions since MongoDB 4.2.

Cheers,
Maxime.

1 Like

Hi @MaBeuLux88 , Thank you for your lovely explanation.
It is weird that we cannot lock database/application between the chunk of code. Sql have this functionality, u can lock all read row.
Thank you for suggesting me capped collection. I read and found that capped collection we can set limit through max:5. But in the case, 5 insert in a day, for the next day if again want to insert 5 more, capped collection will not work.
Anyhow, in the conclusion, I can say, I understood the use and the limitation of transaction. There is no lock we can use to handle concurrent request.

Thank you once again for the fruitful conversation.

1 Like