Reason for the forced default batchsize=1000 in nodejs native driver

I noticed that the node.js driver forces a default batchSize value of 1000 on collection.find, which causes performance issues in my testing.
This overrides the default behaviour of the mongo engine (101 documents, then 16MB batches), I’m wondering what’s the reason.

A bit of context:

  • while doing some testing to evaluate the architectural choices, I found that my data extraction test in node was not very fast.
  • reading about Mongo’s performance, I came across a comparison of the node, java and python drivers, where node was much slower, and decided to run the same test in python, which was consistenly twice faster on a collection.find({})
  • looking at the logs, I noticed that the python driver was setting no batchSize, so the first batch is 101 documents, then a getMore for 16MB (which nearly contained my collection), then the remainder (“normal” mongo behaviour as described there https://www.mongodb.com/docs/manual/tutorial/iterate-a-cursor/#cursor-batches)
  • the node.js driver however sets a default batchSize to 1000, causing endless getMore requests
  • the sources of the driver indeed force a default batchSize (in node-mongodb-native/src/cursor/abstract_cursor.ts at c9a962f15a88efe575abb135165008575efa2572 · mongodb/node-mongodb-native · GitHub)
  • setting a bigger batchSize (around 60000 as that’s what works well in python) improves performance in my tests, but why override the documented behaviour of the engine in the first place?

Thank you for any insight!

1 Like

Hey @Philippe_Smartbiotic,

Given how long ago that default was added, it was likely to try and set a sane default to avoid exceeding the 4MB BSON max document size that existed at that time (increased in SERVER-431 to 16MB).

Our team is currently working on NODE-5906, which may include evaluating the performance implications of removing the default batch size. Assuming we observe a positive impact (:crossed_fingers:) we would consider changing this behavior in a subsequent major release.

3 Likes

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.

Ok, thanks a lot for the info!