Error when bulk upserting a large number of documents (over 3M)

Working with Data Drivers
1

I’m getting this error whenever I insert more than 3 million documents in a bulk insert. I tried executing the bulk insert every 1 million, but I’m still getting the error. I’m running Ubuntu 20.04 and mongodb 5.0.8. Does anyone have any idea? It looks like the buffer is pushing the integer limit in C/C++. Is this the harddrive not keeping up with the lazy writes? Do I need to throttle my upserts somehow?

mongo-c-driver-1.21.0/src/libmongoc/src/mongoc/mongoc-buffer.c:139 _mongoc_buffer_append(): precondition failed: (buffer->datalen + data_size) < INT_MAX

2

I found a workaround. I was doing a bulk execute every so often, but I wasn’t resetting the bulk container object. I didn’t see a bulk.clear() or anything close, so I just reallocated it.

...
bulk.append(upsert_op);
counter++;   
if (counter>1000000) {
  counter = 0;      
  auto result = bulk.execute();
  bulk = voxel_collection.create_bulk_write();
}
...