Increase max document size to at least 64MB

Other MongoDB Topics MongoDB Server Development
26

We use ROS (Robotic Operating System) and it’s distributed between 6 different computers on the vehicle. We have a perception (cameras and lidar), localization/navigation, mapping, control, safety, and a database computer running MongoDB. It’s too heavy of a load for a single computer; we had a team build an amazing (and fragile) Threadripper and it failed. The images come in from six cameras as a 15fps stream of compressed raw stills over the network (not files). Each still from one of the low rez cameras is 5MB and 25MB from the high rez cameras. We thought about writing the raw images (bgr8 format, the binary isn’t a standard format like jpg or png) with a unique id to the filesystem, but that creates different problems. The software isn’t running on the same machine as the MongoDB and when documents are updated or deleted, the filesystem also needs to be updated. It all can be figured out, it just complicates things using the filesystem. It’s much cleaner and faster to write the c++ objects and arrays directly to the database, using lazy writes if the filesystem can’t keep up, and cast them back when read. No encoding/decoding or serialization/deserialization. For example:

To Write:

MyClass my_object;
bsoncxx::types::b_binary b_object
{
  bsoncxx::binary_sub_type::k_binary,
  uint32_t(sizeof(MyClass)),
  reinterpret_cast<uint8_t*>(my_object)
};

Then to read:

MyClass* my_object =
  reinterpret_cast<MyClass*>(my_doc["my_object"].get_binary().bytes);

It’s simple = more stable, less code = faster, and less to maintain = more reliable.

27

Hi, I understand the design thinking behind MongoDB restricting the document size to 16MB. However, I think there should not be any limit on a document size. I think developers should be allowed to structure their data the way they think is best for their product. I think it should just be a situation where companies pay more for their overall database size without any limits on the size of a document. I am currently in the process of considering making changes to the design of my database and it’s quite some work to think about and go through even though we have not even launched our product to the market. I can only imagine what companies that have so much data have to go through to redesign their database and migrate without any issues. It will really be amazing if MongoDB removes the document size limit to make their customers satisfied.

1 Like
28

Ok so I started making changes to my database design and I think the MongoDB limit of 64MB is much better and easier to work with. However, I do understand that it might be strange for older developers to adopt quickly. I only learnt SQL but never worked with it deeply. MongoDB is all I have worked with so making the changes is a lot easier for me and probably other younger developers.

2 Likes
29

There is currently a request for this open on the feedback site.
Anyone interested in seeing the limit increased should probably go and vote for that idea to show their support:

1 Like
30

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.