Duplicate documents get inserted when using insertOne in bulkWrite within transactions

MongoDB Atlas
, ,
1

I am using Node.js and Mongoose to interact with MongoDB. But I encountered a strange error: sometimes (not all the time) duplicate documents get inserted into the collection, even though I just use insertOne in bulkWrite within a transaction, and the createdAt field value is the same for both documents. Please, tell me why?

2

I couldn’t get the whole situation. In your insertOne or bulkWrite, do you insert the same document more than once? If you want to enforce unique documents in a collection, you could use a index with the unique option

3

My query is something like that:

  const session = await mongooseConnection.startSession()
  await session.withTransaction(async () => {
      return TransactionHistory.bulkWrite([
        {
          insertOne: {
            document: {...}
          }
        }
      ], { session })
    })
  } catch (err) {
    if (session.inTransaction()) {
      await session.abortTransaction()
    }
  } finally {
    await session.endSession()
  }

If I set a unique index, I will receive an error: ‘duplicate key’ even if the collection does not contain that key, and the transaction will be rolled back.

4

Please tell me why and how I can fix it?

5

Can you provide a more detailed example? With the documents that are duplicated and the index created in the collection?

6

This is an example of duplicate documents:

[{
  "transactionHash": "0x81c656af8401cd8e36c9b9e69e700bab294931382c2b1174790e9dead94fe085",
  "tokenId": "11000000089572",
  "orderId": 132794,
  "name": "Envia Blade",
  "sellerAddress": "0x5dbf960c5ca99fd92f42b7bc54d2985ce96bff72",
  "buyerAddress": "0xfacf10ed56a87c14a9066a45a5b44abda30a7817",
  "nftType": 1,
  "price": "666.0",
  "floatPrice": 666,
  "currencyCode": "USDT",
  "image": "https://genso.game/nft_images/10000000895.png",
  "nftItemId": "10000000895",
  "type": "equipment",
  "timestamp": 1703725395,
  "createdAt": {
    "$date": "2023-12-28T01:03:43.156Z"
  },
  "updatedAt": {
    "$date": "2023-12-28T01:03:43.156Z"
  }
},
{
  "transactionHash": "0x81c656af8401cd8e36c9b9e69e700bab294931382c2b1174790e9dead94fe085",
  "tokenId": "11000000089572",
  "orderId": 132794,
  "name": "Envia Blade",
  "sellerAddress": "0x5dbf960c5ca99fd92f42b7bc54d2985ce96bff72",
  "buyerAddress": "0xfacf10ed56a87c14a9066a45a5b44abda30a7817",
  "nftType": 1,
  "price": "666.0",
  "floatPrice": 666,
  "currencyCode": "USDT",
  "image": "https://genso.game/nft_images/10000000895.png",
  "nftItemId": "10000000895",
  "type": "equipment",
  "timestamp": 1703725395,
  "createdAt": {
    "$date": "2023-12-28T01:03:43.160Z"
  },
  "updatedAt": {
    "$date": "2023-12-28T01:03:43.160Z"
  }
}]
7

If I set unique for ‘transactionHash’, I will receive a ‘duplicate key’ error, even if the collection does not already contain that key, and the transaction will be rolled back. If I do not set a unique index, I may encounter duplicate documents

8

I tried these two documents with a unique index on transactionHash, and it worked as expected. The first document was inserted with no issues, and for the second one, I got an E11000 duplicate key error collection .... I’m using MongoDB Enterprise 7.0, and the command that I used to create the index was db.test.createIndex({transactionHash: 1}, {unique: true})

9

I am attempting to fix this bug by adding logic to prevent a reentrant call in session.withTransaction() . Here is my code for executing the transaction:

const executeTransaction = async (callback) => {
  const session = await mongooseConnection.startSession()
  try {
    logger.info('Start transaction')
    let isRunning = false
    await session.withTransaction(async () => {
      if (!isRunning) {
        isRunning = true
        logger.info('Transaction is running')       
        const callbackResult = await callback(session)
        return callbackResult
      }
      logger.info('Transaction has already ran')
      return null
    })
    await session.endSession()
    logger.info('End transaction')
  } catch (err) {
    logger.error(`Transaction error: ${err}`)
    if (session.inTransaction()) {
      await session.abortTransaction()
    }
    await session.endSession()
    throw err
  }
}

In a normal case, I receive these logs:
[INFO] 2024-01-16 03:22:21 - Start transaction
[INFO] 2024-01-16 03:22:21 - Transaction is running
[INFO] 2024-01-16 03:22:21 - End transaction
But when I try again, again and again, I get these logs and a MongooseError:
[INFO] 2024-01-16 03:22:10 - Start transaction
[INFO] 2024-01-16 03:22:10 - Transaction is running
[INFO] 2024-01-16 03:22:10 - Transaction has already ran
[INFO] 2024-01-16 03:22:10 - End transaction
MongooseError: Cannot set a document’s session to a session that has ended. Make sure you haven’t called endSession() on the session you are passing to $session().
I checked the database, and the data was inserted without duplicate documents. However, why am I encountering this MongooseError error, and why is the callback function in session.withTransaction() being called multiple times?