Change stream size limit workaround

[Yang_Wu1 (Yang Wu)]

I see there is an open ticket for the 16MB limitation on change stream https://jira.mongodb.org/browse/SERVER-55062

Until it is fixed, given a document can be 16MB as well, change stream can crash when we request the pre-image. Looking at https://www.mongodb.com/docs/manual/changeStreams/#change-streams-with-document-pre--and-post-images, it suggests to

Limit the document size to 8 megabytes

I wonder how can we achieve that with transactions, where we don’t know the full document size until the commit time.

[Stennie_X (Stennie)]

Hi @Yang_Wu1,

@Yang_Wu1

I wonder how can we achieve that with transactions, where we don’t know the full document size until the commit time.

The document size limit applies to each document in a multi-document distributed transaction (MongoDB 4.2+), not the overall size of the transaction.

If individual documents you want pre-images for are likely to approach 8MB or more, I would review the reason for document growth and reconsider your data modelling approach. Large documents are often due to anti-patterns like massive arrays or bloated documents.

Alternatively you could avoid using both post-images and pre-images for a collection which has large documents and consider:

• Requesting only post-images

• Requesting only pre-images in the change stream output and fetching the current document

• Disabling pre-images for collections with large documents

Regards,
Stennie